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ABSTRACT 

The aethods, procedures, and results of an interview 
survey into reactions of user personnel to the Air Force 
Instructional Systea Developaent (ISD) systea are reported. Training 
evaluation, student aeasureaent, and training aanager/instructor 
personnel sere interviewed. All groups regarded the ISD procedures 
favorably, but soae probleas associated with the ISD procedures were 
identified. Recoaaendations for ISD prograa iaproveaent are 
presented. The results of an interview survey of the Air Training 
Coaaand to clarify issues relating to training evaluation and student 
aeasureaent are reported. The survey was also concerned with a 
general assessment of the use and application of the Instructional 
Systea Developaent technique. A total of 139 training evaluation, 
student aeasureaent and training aanager/instructor personnel were 
interviewed. Generally favorable attitudes were reported regarding 
the use and application of the ISD systea, the student aeasureaent 
techniques, and the course review techniques. Problea areas were 
identified and specific recoaaendations for iaproveaent are 
presented. (Author) 
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SUNMARY 

Problem 

The Human Resources Laboratory initiated v three pronged program 
for the purpose of clarifying issues relating to training evaluation 
and student measur^nt. The field survey of the Air Training Comnand, 
reported herein, represents one aspect of this work. The other two 
aspects of the total program involved*. (1) a literature review of the 
progress made in the training evaluation and student measurement tech- 
nology over the period of 1950-1970, and (2) preparation of a user- 
oriented guide to training evaluation and student measurement methods 
and procedures (AFHRL-TR-72-15) . The purposes of the field survey 
were to obtain data that would help to set into perspective current 
Air Force training evaluation and student measurement practices and to 
provide insight into areas needing improvement . Additionally, the Air 
Training Comnand survey was concerned with a general assessment of the 
use and application of the Instructional System Development (ISD) 
technique . 

iHproach 

The semi -structured interview was employed to collect information 
at various ATC centers. Separate interviews were developed for each 
of the four training specialties included in the sample. The speci- 
alties sampled were ISD, training evaluation, student measurement, and 
training managers/ instructors. The sample of respondents was drawn 
from five Air Force Technical Training Centers (lackland , Sheppard, 
Keesler, Chanute, and Lowry) . The sample was composed of 139 respondents, 
divided over the four specialties such that 24 percent of the sample 
were ISD specialists, 15 percent were training evaluation specialists, 
22 percent were student measurement specialists, and 39 percent were 
training managers/ instructors. 

Results 

Each of the four groups thought the ISD procedures achieved a 
different success level in terms of realizing their general purpose of 
graduating students who meet job performance requirements, although 
all groups regarded the ISD procedures favorably. In spite of the gen- 
eral lv favorable attitude toward ISD, a number of problems with the IS) 
procedures were identified by the respondents. These problems were 
concerned with the need for: (1) an information dissemination program 
to clarify misconceptions about the ISD system, (2) streamlined ISD 
procedures, (3) evaluation of each ISD step, and (4) more thorough 
application of the ISD procedures. 
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The general consensus toward the student measurement tecliniques 
employed was also favorable, and the criterion referenced checklist was 
especially believed to be valuable. However, there was evidence tnat: (1) 
some of the written tests employed may lack reliability, (2) standards for 
test update lack adequacy, (3) parallel test forms may be lacking where 
they are required, and (4) increased consideration of' the confidence and 
hierarchical testing approaches is warranted. 

The principal course review techniques employed were graduate, 
supervisor, and student critiques. These techniques were regarded as 
economical and productive of valuable insights into areas for needed 
course revision. Criticisms of these techniques were related to their 
lack of objectivity and the interpretation of the data thev yield by 
persons who are unfamiliar with the total subject matter content or with 
the context in which the recommendations are set. 

Conclusions 



/taong the recommendations for program improvement are: (1) de- 
velopment of an information dissemination program designed to clarify mis- 
conceptions about the 1SD system and to orient training personnel in the 
attributes and goals of the ISD program, (2) increased training for 
personnel involved in ISD use and application (which may also result in a 
decrease in the time required to meet ISD requirements) , (3) development 
of job aids and work methods, along with an increase in communications 
efficiency for the purpose of "streamlining" the ISD procedures, (4) 
evaluation of the proficiency with which each ISD step is performed, 
(5) establishment of guidelines for updating tests, (6) use of alternate 
or parallel tests where required, (7) determination of the predictive 
validity of the tests used, (8) allowing sufficient testing time for the 
purposes on hand, and (9) development of course review techniques that 
can yield longitudinal, objective, and quantitative information. 
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CHATTER I 



INTRODUCTION 



The systems approach to training, although not a recent innova- 
tion, has received increased emphasis over the recent years. AF Manual 
50-2, Instructional System Development, in discussing the background 
and history or the systems approach to training, points out that the 
systems approach involves both curriculum planning (in a logical and 
organized manner, i.e., systematically) and identifying objectives that 
arc to be achieved. 

The formal process used by the Air Force in developing training 
programs within the context of the systems approach, is called Instruc- 
tional System Development, or ISD. The IS) procedure is composed of 
five interacting steps, as shown in Figure 1. As also indicated in 
Figure 1, each step supplies the information needed to accomplish the 
next step, although portions of various steps may often be accomplished 
siirailtaneously. 

The five ISD steps are: 



environment. 

2. Define Education or Training Requirements -- 
determine the tasks that require instruction, 
the level of proficiency to be developed in 
students, and the resources needed to conduct 
the instruction. 

* 

3. Develop Objectives and Tests --identify be- 
haviors (attitudes, knowledges, and skills) 
required for successful job performance and 
construct criterion and enabling objectives, 
as well as achievement tests. 

4. Plan, Develop, and Validate Instruction - - 
select instructional methods, media, and 
equipment that best satisfy learning objec- 
tives; determine the sequencing of the 
instructional material; validate the instruc- 
tional materials to prove that they teach 
what they are designed to teach and to insure 
that all elements of the instructional system 
function effectively in achieving stated 
objectives. 
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5. Conduct ami 1-valuate instruction— identify 
problem areas and corrective actions needed 
in order to satisfy the requirements of the 
operating commands. 

Advantages of Systems Approach 

The systems approach has several advantages, insofar as training 
program development is concerned. The advantages to be gained by 
using a systems approach in training program development are: 

Comprehens iveness - -It ensures that all 
the elements necessary for on-the-job 
performance are included in the training. 

J ob Re lat edness - - 1 1 includes only those ele- 
ments that pertain directly to successful job 
performance as determi. ed by an analysis of 
the system requirements. 

.A ppropriateness - - It emphasizes job behaviors 
wnlch must be demonstrated on the job and 
the prerequisite skills and knowledges that 
satisfy job performance requirements. It 
also specifies the conditions under which the 
behaviors will take place and the minimum 
standard of acceptable performance. 

Practicality- -It includes the concept of 
cost -ef feet iveness so that comparisons 
of desirable features and alternatives are 
completed. 

Flexibility - -It includes posttraining course 
review and student performance analysis. 
This leads to periodic redesign, modification, 
and update of the training program. 

Validity- -The total process ensures that 
the instruction is related to the training 
objectives and the performance requirements. 
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Purpose of Present .Study 

In the fifth step of the TSD model, the actual production of 
trained personnel is accomplished, and the instructional program is 
continuously refined and updated in order to ensure that the training 
program continues to accomplish its goal of producing qualified per- 
sonnel who meet the requirements of the operating commands. Clearly, 
effective techniques for the evaluation of training programs and for 
the measurement of student performance, both during and after training 
are important aspects of this step, and to the ISD procedure as a whole. 
In this sense, training evaluation constitutes the quality control aspect 
of the training system development cycle. Training evaluation, although 
it might occur in one specific phase of training, provides feedback to all 
phases of the training program from the very initial analyses of system 
requirements to the actual conduct of the training. The feedback data 
are required for training improvement, determining if the training meets 
its objectives, and comparing cost -effectiveness of different methods 
and approaches. 

Similarly, student measurement yields information which allows 
the trainer to know whether or not students are meeting course objec- 
tives, and if they need special assistance. Student measurement results 
also find application in the solution to problems related to improved 
student selection and classification, graduate assignment, and advanced 
training. 

Although some type of training evaluation and student measurement 
(however formal or informal) takes place in most training programs, 
there is no universally accepted single best training evaluation or 
student measurement method. There are many issues involved in training 
evaluation and student measurement , and most workers in the field seem 
to agree that training evaluation and student measurement are multi- 
dimensional in nature. 

In order to clarify issues related to training evaluation and stu- 
dent measurement in the Air Force, the Air Force Human Resources 
Laboratory initiated a three-pronged program through a contract with 
the Applied Psychological Services. 

Initially, a literature review was conducted. The literature 
review was concerned with both training evaluation and student measure- 
ment. To ensure complete coverage of the relevant fields, searches 
were completed into the literature covering the period 1950 to 1970, 
including technical reports, journal articles, books, and annual reviews. 
The results of this literature review are documented in the report, 
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Tr aining Evaluation and Student Achievement Measurement: A Review of 
the Literature (AFlfRl-TR-72-3) . The report documents e- 3 describes 
nrogress in and the current state of the technology in the content 
area involved. Current issues and problems are clarified and comments 
made on the offered solutions. 

A second aspect of the approadi to ensuring improved training 
evaluation and student measurement techniques in the Air Force involved 
the development of a report which deals specifically with training 
evaluation and student measurement methods and procedures. It 
summarizes and integrates the evaluation technology state-of-the-art for 
direct application to Air Force systems. Advantages and disadvantages 
of various teduiiques are discussed and, where appropriate, recom- 
mendations are made concerning the application of the procedures to 
specific training evaluation situations. This report is published 
under the title, Some Techniques for the Evaluation of Technical 
Training Courses and Students IAFHKL-TR-7Z-15J . 

Finally, to achieve the goals of the total study, a field survey 
of the Air Force Technical Training Centers was performed. The 
purposes of the field work were to survey and review the training 
evaluation and student measurement practices at the various training 
centers. The methods, procedures, and results of this field survey arc 
the concern of the present report. It was anticipated that the findings 
of such a survey would help to set into perspective current Air Force 
training evaluation and student measurement practices and provide insight 
into areas for needed improvement. As a corollary, a general assessment 
of the use and application of the ISD technique was also included within 
the field survey. 

Collection and Analysis of Field Evaluation Data 

Field evaluations are designed to obtain information that can he 
used by training administrators' for improving and updating training. By 
knowing what operational needs graduates meet and where graduates fail 
to satisfy field requirements, one can determine needed areas for course 
revision. Such evaluations are usually conducted after graduates have 
been on the job for which they were trained for a period of time. There 
are several field evaluative techniques: grad».iate and supervisory question- 
naires, graduate and supervisory interviews, and evaluations of the gradu- 
ates 1 knowledges, skills, and attitudes through tests and rating methods. 
Each possesses its own advantages and disadvantages. 
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The questionnaire, as a self -administrating instrument, can be 
mailed to course graduates and/or their supervisors and similarly 
returned to a central processing office. 

Several areas can be investigated through questionnaire methods. 
One of these is the affective or emotional reactions of graduates to the 
training program. Another is their specific thoughts and suggestions 
regarding training program improvement, i.e., training areas of needed 
additional emphasis or deemphasis on the basis of the graduate's 
experience on the job. 

There are many different types of questionnaire item that can be 
constructed for obtaining evaluative information. The Likert type item 
ask.; the extent of agreement or disagreement with a particular state- 
ment. For example, "Do you (strongly agree), (agree), (mildly agree), 
(mildly disagree), or (strongly disagree) that the training you received 
prepared you adequately for your present job." Another item type is the 
semantic differential item. Itere, bipolar adjectives are used to define 
the eni'xunts of semantic dimensions. An example of a semantic differ- 
ential i :em is: "Place a checkmark anywhere along the line, between the 
pair of words, to indicate your feelings about the training program." 

Another popular questionnaire item type is sentence completion. 
The stan of the sentence, which is provided 1 to the respondent, channels 
his thoughts in a particular direction. For instance, "One of the things 
that is done differently on the job from the way it was taught in school 



Miltiple choice questions frequently appear in questionnaires. If 
the response category "other" is included along with the response cate- 
gories, the changes of losing data are decreased. An example of a multi- 
ple choice item is: 

Check the one area of training which was most irrelevant 
to your present job: 

1. theory of electronics 

2. use of test equipment 

5 . component identification 

4. circuit design and troubleshooting 

S. other, specify 
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Forced-choice items consist of a group of statements which are 
matched on favorability. In a two-choice question, one of the items will 
differentiate in terms of one characteristic, while the other item will 
differentiate in terms of another characteristic. The statement chose by 
the respondent reflects his preference. Forced-choice items can have 
three four, or five choices as well. One variation of t-iis item type 
is to have two desirable statements and two undesirable statements, with 
the respondent selecting the one which most represents his point of view 
and the one which least represents his point of view. This tectaique is 
held to eliminate much response dishonesty. A forced-choice item might 
appear as follows: 

Place an "M" next to the option that best describes your 
feelings to the set. Place an "L" next to the one which 
least describes your feelings: 

a. The course prepared me for adequate function 
on the job. 

b. The tools and equipment in the school aren't 
much like those on the job. 

c. The laboratory exercises were much like the 
work we do on the job. 

d. Nfeich of what we do on the job is different from 
the way we were taught to do it in school. 

The second major technique used for collecting field data for 
training evaluative purposes, the interview, consists of three basic 
types: unsystematic, systematic, and standardized. In the unsystematic 
interview, usually no particular sequence is followed, although the 
interviewer may have a set of questions he developed in advance. This 
approach can prove to be highly unreliable and invalid. In the systematic 
interview, the interviewer has a plan and format. However, within the 
plan the interviewer is allowed the flexibility to probe where he thinks 
it is necessary. The standardized interview is characterized by a 
series of questions that are asked by all interviewers of all inter- 
viewees in the same sequence and with the same wording. The technique 
eliminates the interviewer as a source of response variability. 

Finally, training evaluation data can be collected in the field 
through various testing and rating procedures. Performance testing, in 
which the performance of the graduate on various job aspects is observed 
and scored, represents the procedure used most widely. 
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The scoring is based on adherence to correct procedures, adherence 
of a final product to prescribed standards, care and use of tools, 
observance to safety precautions, and time for task completion. Typically, 
the scoring form contains all the sequential elements, which must be 
performed correctly, in a task. The scorer observes the examinee's per- 
formance on the task and scores each subelement accordingly. A final 
score, the sum of all subscores, is usually obtained from this type of 
test. Analysis of the test scores yields the data required for training 
program improvement. 

The proving ground represents an advanced adaptation of the per- 
formance test situation. Here, the graduate is cycled through all the 
tasks of a job in a short period of time. As he performs on each task, 
the graduate is evaluated. lie, in turn, evaluates the training he 
received in relation to the task. 

It is possible to build criterion referenced methods into the super- 
visory questionnaire or the performance testing field evaluation paradigm, 
when a graduate is evaluated through a criterion referenced technique, 
the graduate's performance is compared with an absolute standard to de- 
termine if he attained acceptable (criterion) performance. The absolute 
standard may be a performance test item of a given level of difficulty 
or achievement of an absolute score. The characteristics of criterion 
referenced scores are that they: 

1. indicate the degree of competence attained, in- 
dependent of the performance of others 

2. measure performance with regard to specified 
absolute standards (criteria) of attainment 

3. minimize individual differences 

4. consider variability irrelevant 

Generally, the criterion referenced score indicates how well a 
person performs with regard to a specified standard of behavior. Cri- 
terion objectives, which arc expressed in terms of specific behaviors, 
must be identified. The criterion objectives specify the operations and 
knowledges the graduate must demonstrate to satisfy a job performance 
requirements. Individual differences among examinees are considered 
irrelevant, since the examinee is graded against a single external stand- 
ard rather than against other job incumbents. Criterion referenced 
measurement programs usually use S (satisfactory) and U (unsatisfactory) 
grades . 
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The criterion referenced approach is desirable in those situations 
where an indication is wanted of absolute proficiency or ability of the 
graduate to perform on a particular job. Since an important concept un- 
derlying technical training is that of training students to a defined 
standard as represented by the job requirements, it appears that the 
criterion referenced approach can be most appropriate relative to skill 
training. 

Several sequential steps are involved in the development and 
administration of criterion referenced instilments . 

1. Following a job analysis, the criterion job be- 
haviors (tasks) are identified on which a gradu- 
ate must be able to demonstrate conyetency . 

2. Tests are developed, one for each criterion be- 
havior (task), to measure competency. 

3. "Minimally acceptable" and "desirable" cut scores 
are established for each test. 

4. The tests are administered and the graduates are 
evaluated relative to their performance as com- 
pared with the cut scores. 

If a supervisory evaluation rather than a performance testing 
approach is followed, then rating scales are developed for each criterion 
behavior. The supervisor of each person being evaluated rates the per- 
formance of each graduate in relationship to each of the cut points. 
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amTim ir 

METHOD 

The semi -structured interview method was employed to collect 
the information required from the field survey. The semi -structured 
interview has several advantages over other methods of collecting field 
data. This technique ensures that all relevant areas are covered by 
appropriate questions, but permits retaining the flavor and details of 
responses by recording them in the interviewee's own words. The method 
also allows for probing by the interviewer in order to gain more insight 
into areas mentioned by the interviewee. Whenever the interviewer 
considers it to be necessary, the interview procedure can be varied to 
meet individual cases. 

The face-to-face nature of this type of interview ensures that 
the interviewee will de/ote his attention to the questions asked, and 
that he will respond. The interviewee is likely to feel quite comfort- 
able and free from stress with this technique and, accordingly, is 
likely to be quite willing to amplify his responses when necessary. 

As compared with an interview, a printed questionnaire is less 
likely to elicit frank statements. The semi -structured interview, with 
its informal atmosphere, is most likely to lead to a situation in which 
the interviewee will be candid, and the lattitude allowed the interviewer 
can be used for establishing cooperation and rapport. 



Development of Interview 

Prior to developing the interview, an attempt was made to acquire 
a better insight into the requirements of the training evaluation 
student measurement, and the ISD program as they apply to the Air' 
Technical Training Command. Various Air Force manuals were reviewed. 
This review included, but was not limited to, the following: ATM 50-9- - 
Principles and Techniques of Instruction. AFM S0-2- - Instruc tional 
system U^elopment ATtR -Tr aining Evalua t ion and Cours e 

neviews, aiur ^-- Measurement . ATCR !>M6-- Stu3enrKecord of Traini ng. 
AlHTTI- 26-- Student S^eduling : and Administration . A'll'k 52-29- -Shttkn*-* 
Critique Program, aitjr 52 -i« - Training Materials , and ATCR 50-4 -lnstru c 

tor Supervision. Other documents, mainly released as project reports 

by various ATC zenters, were also reviewed. Further insights into the 
requirements were gained through conferences and discussions with per- 
sonnel from the project sponsoring staff and the ATC staff. 
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In developing the interview, the areas of interest, i.e., those 
areas into which the interview would probe, were first identified. 
These areas were: 

• overall evaluation of the TSD program 

• evaluation of the specific ISD steps 

• general opinions about ISD and its application 

• effectiveness of the IS) steps for achieving 
their purposes 

• attitudes toward ISD 

• applications of the ISD principles 

• specific problems inherent in IS) 

• methods for improving ISD 

• how ISD was used and how it was helpful 

• evaluation of the training evaluation tech- 
niques used 

• problems associated with the training eval- 
uation techniques employed 

• how information from the training evaluations 
was fed back tj the user 

• how the user implemented this information 

• evaluation of the techniques used to measure 
student achievement 

• problems associated with the student measure- 
ment techniques used 

• how results of the student measurements are 
used 



Initial Assembly of Interview and Pretest 

The interview, in its original form, consisted cf over 100 items 
and included several different data collection techniques. These 
included the critical incident approach, rating scales, rank ordering, 
magnitude estimations, and open-ended questions. 

This interview was pretested on a sample of seven specialists 
working at the 3415th TS, LTTC, Lowry AFB, Colorado, in specialties 
which were to be included in the eventual sample. The pretesting indi- 
cated one major required change, aside from the typical editing and re- 
phrasing of questions for the purpose of enhancing clarity and under- 
standing . 
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Since the interview inquired into three major content areas (ISD, 
training evaluation, and student measurement) questions were asked of 
the pretest respondents in each of these areas. Yet, those respondents 
who were questioned on issues outside their immediate concern (either 
ISD, or training evaluation, or student measurement) were often unable 
to provide knowledgeable responses. Or, if responses were offered, 
their accuracy or meaningfulness was suspect. In addition, the length 
of the interview was unwieldy—it required in excess of two hours. 

As a result of the pretest, the original interview was reorganized 
into five separate interviews. Thus, respondents would receive questions 
that were specific to their background and experience. The fractionation 
of the original interview into five separate interviews reduced the time 
required to complete an interview considerably, although there were many 
questions that appeared on more than one interview form. Whereas origi- 
nally over two hours were required for the interview, now the interviews 
could he completed in one-half hour to one and one-quarter hours, depend- 
ing on the interview form. The different training specialties into 
which the interview was divided were training evaluation, student meas- 
urement, training managers/instructors, ISD, and headquarters. 

Final Interviews 

The interview instruments utilized are presented in the Anpendix 
to this report and are described below. 

At the start of the interview, but after the initial introduction, 
explanation of the study, and the purposes of the interview, the inter- 
viewee was provided with two handouts. These were the flow diagram of 
the ISD model, as presented in Figure 1, and a set of definitions of the 
five ISD steps. The ISD definitions are presented as Figure 2. These 
were provided so that if the interviewee wished to re familiarize himself 
with the concepts, he could, and those interviewees whose knowledge and 
familiarization with ISD was spotty could refer to the handouts whenever 
necessary during the interview. 

Certain questions in the interview required the interviewee to refer 
to specific definitions or scales of values. Wherever this was required, 
handout cards were displayed. These are contained at the end of each 
interview set in Appendix A. 
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ISD DEFINITIONS 



\nalv:c Svstem Requirements (Step 1) : Familiarization with the 
operational system, the major components ^tooim^,** 
understanding the system's mission. Identification of the job (s) to 
be performed; placement of each job in perspective with objectives, 
requirements, and the overall environment of the operational system. 

Define E ducation or Training Requ irements (Step 2); Generation 
of an inventory of job periormance requirements ana tne required 
levels of performance. Determination of the following: number of per- 
sonnel to be qualified and methods for acquiring 

to develop the instruction, criticality and learning difficulty of tasks, 
availability and qualifications of personnel and resources needed to 
develop and' implement the instructional system, and leadtime for 
development and installation. 

Develop Ob jectives and Tests (Step ?) : Identification of know- 
ledges, skills, and attitudes required i or successful performance. 
Development and construction of criterion and enabling objectives and 
tests to evaluate student achievement and progress and to indicate 
effectiveness of the instructional system. Construction of aptitude, 
diagnostic, survey, and pretests. These tests may be used to determine 
the knowledges, skills, and attitudes the prospective student has m 
his repertoire. 

Plan, rwelon. and Validate Instruction (Step 4) : Involves se- 
quencing learning activities in an order that produces the required 
?e*3 in the ihortest time. Selection of the instructional methods , 
media, Find equipment. Development and validation of instructional 
materials. Validation indicates if instructional materials teach that 
which they are designed to teach and insures that all elements of the 
Instructional system function effectively. 

Conduct and Eva luate Instruction (Step 5V Determination of 
student attainment o! learning objectives/ Thia often identifies 
problem areas and the corrective action needed. Insures that the 
product of the instruction system (i.e., the qualified personnel) meet 
the requirements of the operating commands. 



Figure 2. Definitions of the five Instructional System Development steps. 
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Regardless of the content, each interview first acquired routine 
demographic information. This information involved the interviewee's 
name and current assignment, his time in his current assignment, and 
an ATC course with which the interviewee was familiar and that had 
been developed in accordance with the ISD principles. The reason for 
the last item was that the interviewee was asked later about techniques 
and experiences in relationship to a specific coutsc with which he was 
familiar. If the interviewee was not familiar with a course that was 
developed in accordance with the ISD model, then the course with which 
ho was most familiar was identified and referred to during the interview. 

Hie additional content of each of the four ISD interviews is 
described categorically below: 

ISD Specialists 

• familiarity and depth of experience with ISD 

• the amount of emphasis placed on each ISD 
step in developing the course and if the step 
was not emphasized, why 

• the success of certain ISD steps in developing 
the course 

• how certain steps were helpful in developing 
the course 

• identification of a situation where ISD steps 
were used hut were not helpful in developing 
the course 

Training Kvaluation Specialists 

• the frequency and method of feedback for course 
revisions from course review and field evalua- 
tion techniques 

• evaluation of course review and field evaluation 
techniques 

• problems with the course review and field eval- 
uation techniques 

• course review and field evaluation techniques 
that could be used to identify areas of over and 
undcrt raining 

• identification of situations in which course review 
and field evaluation techniques were used to modi- 
fy the course 
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Student Measurement Specialists 

• types of student achievement measurement tech- 
niques used 

• end of block written tests- -evaluation, problems, 
determination of failing grades, recommendations 

to failing students, type of item analysis conducted, 
relationship to other measures 

• performance tests- -evaluation, problems, deter- 
mination of failing grades, recoflmendations to 
failing students 

•criterion checklists- -evaluation, problems, de- 
termination of failing grades, recommendations 
to failing students 

• other student measurement techniques that could 
be used 

• feedback of student measurement results to course 
developers and how they were used 

• identification of a situation where student measure- 
ment results were used to modify the course 

• the frequency with which student measurements 
were reviewed and the frequency with which they 
were modified 

• identification of a situation where student critiques 
were used to modify the course 

Training Managers/Instructors 

• the frequency and method of feedback for course 
revisions from course review and field evaluation 
techniques 

• evaluation of course review and field evaluation 
techniques 

• problems with course review and field evaluation 
techniques 

• course review and field evaluation techniques that 
could be used to identify areas of over and under- 
training 

• identification of situations in which course review 
and field evaluation techniques were used to modify 
the course 

•types of student achievement measurement tech- 
niques used 

• end of block written tests—evaluation, problems, 
determination of failing grades, recommendations 
to failing students 
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• feedback of student measurements to course 
developers and how they were used 

• identification of a situation in which student 
measurement results were used to modify 

the course 

• identification of a situation where student 
critiques were used to modify the course 

• criteria for judging instruct ional effective- 
ness 

• identification of the best and worst aspects of 
ISD and ranking them 

• reasons for supporting the IS) program 

• problems in developing courses using the ISD 
techniques 

• evaluation of ISD techniques 

In addition to the interview content, as outlined above, there 
were a series of items which pertained specifically to the ISD pro- 
gram and were included in all interviews. These items involved the 
following concepts: 

• evaluation of the ISD program 

• suggestions for improving the use of ISD 

• reasons for avoiding ISD 

• problems in using ISD 

• evaluation of ISD steps for achieving an ef- 
fective instructional program 

• evaluation of ISD steps for the amount of em- 
phasis given to each in developing the course 

The final question in every interview, "Is there anything else 
you would like to tell me?" gave the interviewee the opportunity to 
free associate and in some cases to discuss aspects of trais.ing and 
ISD that may not have come out in the earlier parts of the interview. 
Some interviewees took the opportunity to clarify and elaborate on 
points that were made during the interview, and others discussed 
related subjects of interest and concern. 

Reliability of Intervi ew 

Reliability, in terms of consistency of response, was investi- 
gated through inclusion of parallel items within each interview, i.e., 
items shich queried into the samo issue but were stated differently 
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than their counterparts. Each of the five interview forms contained 
at least two pairs of items that were analyzed for response relia- 
bility. Examples of the reliability items are: What specific or 
general suggestions do you have for improving the use and application 
of ISD? and What one improvement would help ISD utilization most?; In 
general, would vou rate or describe the ISD program as 'superior, 
"verv good," "good',' "fair," or " poor" for achieving the stated 
objectives of turning 5uF"graduateT^io meet job performance require- 
ments efficiently and effectively? and How would you evaluate the 
ISD program, on the basis of the values 0 to 100, for achieving its 
purposes? 

Interviewer Training 

Three interviewers collected the data in the field. Hie inter- 
viewers were all trained in psychology and the principles and tech- 
niques of structured and unstructured interviewing. Two interviewers 
were from the staff of Applied Psychological Services; the third was 
the project monitor. Training sessions were conducted with the 
interviewers. In these sessions, the purposes of the study, the 
areas of concern and interest, and the interview were discussed. All 
instructions to the interviewer, as they occurred in the intejroew, 
were reviewed, along with the purpose and meaning of each item. The 
manner in which the responses were to be recorded on the response 
sheets was covered. Other topics of discussion during the interviewer 
training sessions were: appropriate tunes to probe, methods tor 
probing, how to handle reluctant interviewees, and organization and 
management of Air Force training courses. 

Also included within the training sessions were considerations 
of typical problems inherent in the semistructured interview such as . 
interviewer bias, wording and posing questions, assuring the respondent 
that his responses would be used for research purposes only and would 
never be attributed to him, and methods for conducting a postinterview 
debriefing with each respondent. 

Additionally, all interviewees 
with the reference documents which are listed on page 16 of this 

report. 
Sample 

The sample of interviewees included personnel from five different 
organizational or specialty areas. The sources from which the sample 
was drawn were: Instructional System Development, Training Manager/ 
Instructor, Student Measurement, Training Evaluation, and Headquarters. 
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Six primary sampling units were involved. These included five 
Air Force Technical Training Schools and Headquarters, Air Training 
Command (ATC) at Randolph Air Force Base, Texas. The five technical 
training schools are located at the Lackland, Sheppard, Keesler, 
Chanute, and Lowry Air Force Bases. The various specialists enumerated 
above were available at each of the five technical training centers, 
while headquarters personnel were available at Headquarters, Air 
Training Command. 

Within specialties and centers, a random and proportional samp- 
ling approach was followed. This assured representation within the 
final sample of each specialty in proportion to its relative repre- 
sentation at each technical training center. Thus, each stratum 
represented in the total sample reflects its relative proportion in 
the population at each Air Training Command site. Table 1 presents, 
by job specialty and by Air Force Technical Training Center, the 
number of interviewees in the final sample. 
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CHAPTER III 



RESULTS 

Description of Interviewees 

The introductory aspect of each interview inquired into the back- 
ground and experience of each interviewee. The initial item inquired 
into the length of tin© the interviewees were in their then current 
position. Figure 3 represents the mean percentage of each subsample 
by years in the current assignment for the four training specialties 
sampled. The fifth job specialty, headquarters personnel, was not 
included in the analysis because the headquarters sample consisted of 
only three interviewees. Figure 3 indicates that for each of the four 
specialties, the majority of respondents had at least from one to 
three years of experience in their current assignment. A very low 
percentage of respondents had less than one year of experience. The 
range of this percentage of respondents with less than one year on 
their current job was from S.9 per cent to 15.6 per cent. The ISD 
job specialty had the largest percentage of respondents with less than 
one year on the job (i.e., 15.6 per cent). Since ISD is a relatively 
new specialty, it might be expected that a greater percentage of these 
specialists would possess minimum on-the-job experience. 

Medians for the number of years in current assignment were also 
calculated. These were: SM* --2.4 years, TE* --4.2 years, TM/I* --2.4 
years, and ISD* --2.1 years. With the exception of he TE group (which 
had a median of 4.2 years), the median values for t - other three groups 
are very similar, with 2.1 to 2.4 years on the job representing the 
range of medians. 

Since the interviewees were questioned about their opinions of the 
ISD program, an indication of their direct experience with ISD was also 
sought in the introductory questions. Figure 4 presents the percentage 
of each specialty in the sample that was familiar with a course developed 
using the principles of the ISD model. All of the ISD specialist interview- 
ees were familiar with a course tliat was developed under the ISD program. 
Approximately three-fourths of the student measurement and training eval- 
uation specialists were familiar with a course which was so developed. 
However, among the training managers/ instructors sampled, only 41 per 
cent were familiar with such a course. In terms of the entire sample 
(across job specialties), 65 per cent were familiar with courses which 
had been developed in accordance with the ISD model. This overall per- 
centage seems low in view of the advocated widespread use of the ISD 



T »!-Student Measurement, TE -Training Evaluation, TM/I -Training 
Managers/ Instructors, and ISD- Instructional System Development. 




YEARS IN CURRENT ASSIGNMENT 



Figure 3. Mean time in current assignment for sampled interviewees 
four training specialties. 
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system within the Air Force and suggests that many courses or course 
elements have been developed on some other foundation. 



JOB SPECIALTY 
ISO 



SM 
TE 
TM/I 



100% 


75% 




70% 






41% 









Figure 4. Percentoge of each subsample possessing familiarity with 
courses developed in accordance with ISO principles. 



Instructional System Development Specialists 



Thirty-three ISD specialists were interviewed at the five Air Force 
technical training schools. The ISD interview consisted of two main 
sections. The first section of the interview dealt with the specific 
application of. the ISD steps. The first substantive area that was 
investigated in this section of the interview was the amount of emphasis 
that was placed on each of the five IS) steps in developing the course 
with which the interviewee was most familiar and which had been developed 
in accordance with the ISD model. Then, the degree of success of two 
(the first and last) IS) steps for achieving their purpose was examined. 
The amount of emphasis and the degree of success were quantified through 
employment of the magnitude estimation technique (Torgerson, 1958) . To 
™J Jr. scales ranging from 0 to 100 were presented to the interviewees 
with the instruction that they select a value representing the amount of 
emphasis on each of the five steps and the degree of success for each 
of two steps. 



28 

33 



lanphasis on Each ISP Step 

Items 4 11. 14, 17, and 20 of the ISD interview (Appendix A) 
pertaineTto'the emphasis placed on each ISD step during the development 
Sf a c^se whichlSd been developed through the ISD system approach. 
?he means and standard deviations of the reported emphasis placed on 
each 133 step are presented as Table 2. 



Table 2 



Amount of Emphasis Placed on ISD Steps in Course Development 



1. Analyse System Requirements 

2. Define Education and Training Requirements 
3*. Develop Objectives and Tests 

4. Plan, Develop, and Validate Instruction 

5*. Conduct and Evaluate Instruction 





Standard 


Mean 


Deviation 


69 


26 


71 


24 


89 


14 


80 


21 


77 


27 



Table 2 indicates that, according to the scale of emphasis, the 

last three ISD steps tended to receive ^^ f «^%^ e ^ive ly 
two. Steps 1 and 2 had mean emphasis values of 69 and "l^esp^tiveiy, 
while the mean values assigned to the remaining steps ranged from 77 to 
$ The sSard deviations for all but the third step (develop objec- 
tives atd tests) seem sizable. For step 1 (analyze system requirements) 
thole respondents who selected low e^hasisvalues did so because the 
analysis of system requirements was performed outside the training 
department. In these cases, the course developers used either occupa- 
tional specialty reports, expert opinions, and judgments, or data that 
were used for previous courses. On the other hand those respondents 
who selected high emphasis values claimed: "step 1 vas basic to the 
ISD program and therefore very important," "required great effort to 
complete," and "took a long time to do." 

The respondents who placed little emphasis on step 2 (define 
education and training requirements) did so ^^%^ e L^ t ?i?Sd 
insufficient time, used job documents for defining the education and 
training requirements, or did not have to perform the details of the 
step because they were accomplished by other departments. 

The interviewees who indicated that extensive emphasis was placed 
on step 3 (develop objectives and tests) offered very similar 

to those offered for step 1. Some interviewees clai^ that most of 
step 3 was accomplished while doing step 2 and 1 or that some new 
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courses were revamps of existing courses. In these cases, step 3 did 
not require too much emphasis. A few respondents indicated that the 
step 3 had been performed to satisfy regulations and/or requests made 
by their superiors. This suggests that, in these cases, step 3 was 
emphasized more than it would have been if the respondents had not 
been so directed. The most common reasons for step 3 receiving an 
extensive amount of emphasis was: "it is important," and "required if 
there is to be transfer to the field situation." 

There were no respondents who claimed that very little emphasis 
was placed on step 4 (plan, develop, and validate instruction). Five 
respondents claimed that a moderate amount of emphasis was placed on 
step 4, whereas the rest of the respondents indicated that an exten- 
sive amount of emphasis was placed on this step. Respondents who 
placed a moderate degree of emphasis on this step indicated reasons 
such as: "the step is incomplete," "used material from previous 
course," and "not enough time allowed." The responses from the high 
end of the emphasis scale were, again, similar to those responses 
offered for the other ISD steps. 

In regard to step 5 (conduct and evaluate instruction) some of 
the ISD specialists indicated that this activity was actually the 
responsibility of instructors and other sections. Accordingly, they 
placed relatively little emphasis on the step. Others indicated that 
the fifth step of the IS) model required much emphasis since it "is 
the crux of a good training program," and "since ISD is new, we had 
to be extra cautious." Other responses at the high end of the scale 
indicated that student critiques and field evaluations were constantly 
being reviewed for purposes of evaluating instruction, and these 
required a considerable amount of consideration and effort. 

Finally, some of the ISD specialists replied that since they 
were responding in terms of the ISD based course with which they were 
most familiar and since the course involved was currently under devel- 
opment, there was little basis for their estimate in regard to step 5. 
This may account for the somewhat elevated standard deviation for 
this step. 

S uccess of Each ISD Step 

As was stated earlier in this section, the interviewees were asked 
to provide a magnitude estimate of the success achieved in implementing 
ISD steps 1 and S. No inquiries into the success of steps 2, 3, and 4 
were made because the interview pretest results indicated that the 
respondents assigned a very similar success value to each step. It 
seems that, to some extent, an overall halo effect was functioning in 
the pretest evaluations. This tendency was also inherent in the field 
survey, where the same mean success value (and very similar standard 
deviations) was obtained for both ISD steps investigated (Table 3) . 
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Table 3 

Amount of Success of ISD Steps 1 and 5 

Standard 

ISP Steps Mean Deviation 

1 . Analyze System Requirements 84 17 

5. Conduct and Evaluate Instruction 84 ^ 



The mean value of 84 indicates that both steps 1 and 5 «• 
considered to be extremely successful for ij^lementing^ controlling 
course effectiveness. Some of the reasons why ^ responds con- 
sidered step 1 to be successful are contained in thejtoll«jlng typical 
responses: "helped to remove unnecessary material from the course, 
Evaluation ofthe entire course indicated that the course was 
successful and therefore step 1 had to be successful, and it brought 
about a closer alignment with the needs of the field." Step 5 was 
considered to be successful because: "it resulted in unproved 
instruction." "it reduced course time making the course more cost- 
effective™ and "by implication since the course worked well step 5 
had to be successful." 

Why Steps 1 and 5 Were Helpful 

Item 8 of the interview asked how the first ISD step, analyze 
system requirements, was helpful in planning the course. Generally, 
the Stewiewees attributed most of the benefits derived from this 
step to result from the hierarchical ranking the .analysis 
allSwL according to the interviewees, more precisely defined course 
JliSt, better identification of areas of emphasis, and courses which 
are structured so as to meet the job requirements . 

The same question was asked in regard to step 5, conduct and 
evaluate instruction. Here, the interviewees indicated that this step 
had identified changes needed in the lesson plans, changes needed in 
time allocations, and changes in training methods or emphasis. 

In no case could a respondent recall a situation in which steps 
1 or 5 were used in developing a course and the course was not bene- 
fitted Items 10 and 26 of the interview inquired specifically into 
this point, but were unproductive in that "no response" was recorded 
for all interviewees. This suggests that the application of the ISD 
principles to course development has met with considerable success 
wherever applied, as seen by over 30 ISD specialists assigned to the 
five Air Force technical training schools sampled. 
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General Opinions 



The second main section of the interview with the ISD speci- 
f*"!*,* 18 concerned with general opinions toward the ISD model, its 
Ttl 32 ?S! Ucatlon - <? Init ;? u y. the interviewees were asked to evalu- 
5£w. ^ progn "? l° r a< X hievin S the sta ted objectives of graduating 
ti^?v ?£° nieet 1 J0b ^formance requirements efficiently and^ffec? 
in Table J* * t0 the IS ° evaluation "a. (item 27) are prated 



Table 4 

Results of Evaluation of the ISD Program by ISD Specialists 
. (N ■ 33} 

Classification Per cent of Respondents 

Superior gi 
Very good 30 



Good 
Fair 



6 
0 



Poor 0 
No response 3 



ui^ m i en l ly rir} aTge m i° Ti tY of the ISD specialists thought very 
L??£^ e & progran \ s i*ty-one per cent claimed thVprogr^ 
Snt nTtlll" in UmS ° f ? c |? ievi ng ^s stated objectives, 91 per 
cent of the responses were either "superior" or 'Very good." 

*!? a Parallel question, each respondent was asked to evaluate 
the ISD program along a numerical scale which ranged from 0 to 100 
The evaluation was in terms of the extent to which the program achieves 
tfii'SFT" r? C Mean °V hi ? ^tion was 78, with HtandaS dlv!a- 
iJSi °f i ' ^ "5?" Vall f 15 W* 111 / concordant with the categorical 
SSi CS J 1S Sf Sed ab0Ve ' (If one assumes each of the five categories, 
discussed above to represent a 20 point range, then the expected vllie 

78 901? nB Rl tlJ ? e "'""IS if [C61 X 903 * ( *° x 70 > * C6 x 50)/100 - 
78.90) J. The close accord between the responses to these two Questions 

ttlnJe^ie^rl^ons 5 ^ 5 8 C0I,tentl ° n faVOring ^ reliabili tT of 
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Problems Inherent in ISD Technique 

The next area of investigation pertained to the problems that 
were mel bTthe respondents in using the ISD model and suggestions 
Sat the interviewees thought might be employed to improve the jppli- 
cSion^nd utilization oflSD. The most frequently mentioned problem 
wfs concerned with the lack of adequately tja^^sonneU This 
implies both a need for more personnel and for training these persons 
in ISD methods and techniques. 

A second frequently mentioned problem was insufficient time to 
complete tnfvartoul ISD steps. This problem arises as a^ult of^ 
several factors. For some course developers, ^.^f^^^Je 
unrealistic deadlines (for the completion of individual ISD steps ) were 
s^so that themost proficient course developer ^ouldnot fini^^ ^ 
tL allowed. Other course developers may not have been ^to^meet 
deadlines due to a lack of training in IS). Still other course oevexopcia 
my no? havf had sufficient personnel or technical support available. 

A third set of problems was concerned with a variety of charac- 
teristics which might be anticipated in any evolving program. For 
example, several respondents pointed to insufficient tods as titt 
Jaurce of their difficulty with the ISD program. Others mentioned 
Kuate e^uipmlnt (e.g 5 ., training aids/devices, training facilities 
and documents). 

These oroblems are real to the ISD specialist and, in some cases 
can ecl^he^Jositive attributes of t^I^p = Accordingly, 
cniiitinns tD these problems seem required if the IS) program is zo 
^ ™11?. ^s^lies the development of a ^ ^roughandcom- 
Xte training program in the use and application of IS), ^^T^ 56 
in tL number of trained ISD specialists, and increased information 
dissemination regarding the ISD program. 

Other suggestions offered by the respondents included: (D re- 
oerforminc or^erforming steps 1 and 2 for existing courses, (2) de- 
vl^InfLreening andleleSion criteria for ISD specialists, and (3) 
improving the ISD documents so that they are more readable. 

A related question asked why people avoid using the ISD , me r t ^°^ T . c 
Here the respondents seemed to think that: (1) many course developers 
Sr eS P^ia!fT^ oldeTSnes are more comfortable and secure performing 
iTtheif accustomed manner and they, accordingly, resist ctogej?) 
Sere is too much ignorance regarding ISD, and (3) course development 
through the use of the model requires too much time. 
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Training Evaluation Specialists 

IWnty-one training eva Ira tors were interviewed for the purpose 
of obtaining information in two principal areas: (1) ISD methods and 
procedures, and (2) training evaluation. 

ISD Evaluation 

Jte training evaluation specialists were asked to evaluate the 
global ISD program on a categorical scale which ranged from "poor" 
to "superior." This evaluation was made, as for the ISD specialists 
in terms of how well the ISD method achieved its goal of graduating 
students who meet job performance requirements efficiently and 
effectively. Table S presents the result of this evaluation. 

Table S 

Results of Evaluation of the ISD Program 
by Training Evaluation Specialists (5 ■ 21) 

Classification Per cent of Respondents 

Superior 10 

Very good 43 

Good 29 

Fair 14 

Poor 0 

No response 4 

A small percentage (i.e., 10 per cent) of the training evaluators 
rated the ISD program as "superior" and 43 per cent rated it as "very 
good." The remaining responses fell in the "good" and the "fair" 
categories. Here, there is a lower percentage of respondents employing 
the two higher categories than for the ISD specialists. This trend 
may nave been anticipated jince the ISD specialists, in view of their 
more total personal involvement in the first four steps of the ISD 
sequence, would be expected to provide a more favorable self- evaluation. 

When the training evaluation specialists rated the ISD program 
(for achieving its purposes) on the quantitative scale, ranging from 0 

L il m 12 ' th f ^V**™* ^an rating was 58, with a standard devia- 
™? • J?" j ^\ val ™ 1S similarly lower than the corresponding value 
(78) yielded by the ISD specialists. (The anticipated magnitude estima- 
tion value for the training evaluators on the basis of the replies to the 

a4 e ^30)/100 UeS 5 1 ° n j3 WaS CC1 ° X 9 ° ) + C43 X 7 ° } + (29 X 50) 
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rroblems Inherent in ISP Technique 

The training evaluation specialists also most frequently men- 
tioned ?he lac^f ade^ately trainer personnel as the major problem 
w ?ntn tiS » technique. tL other . frequently menuoned 
with ISD were that: (1) it is extensively time consuming, (2) ^cuoa 
tional survey data m not readily available (i.e., the job analytic 
da?a^ulred in step 1 of the model) for updating a course, and (3) 
^re^ alack of coordination between the IS) and training evaluation 
efforts with respect to course evaluations. 

Inquiry into solutions for these problems resulted in suggestions 
such aT increasing the accessibility of the IS) course to ISD special- 
ists % a^to increase the number of trained specialists at the training 
centers ,^nd the development of courses in the concepts and benefits of 
ISD for personnel associated with the Air Training Command, but net 
aSiSed^o IhelSD specialty. It was also suggested that streamlined 
prlcedures be developed for course development under the ISD P^iples. 
The streamlined procedures would result in \^^* m ^Jf* CO[irse 
needed to prepare a course. The total time needed for ISDing a course 
could also P be reduced, as the respondents suggested, by increasing the 
manpower assigned to developing the course. 

Other considerations for improving ISD pertained to tte analyses 
of system requirements that are performed m th f, ^"^f^^f^ 
model and ^information dissemination program which would help to 
^sitivelv motivate people in managerial positions m regard to the 
^ prclram £e forme? consideration, that of developing a J* anal- 
ysis is crucial to successful course development under the ISD 
£e^ts. In some cases, the job analyses were m* available, and 
existing data and/or ad hoc analyses were 

centers or in conferences, rather than in the field. The latter 
consideration, inforoation dissemination and diffusion jas not offered 
in the sense of propagandizing or attributing qualities to I^that^e 
not factual. Rather, the intent was to inform and educate people about 
?ne ISD £thods, how they are used, and the inherent advantages of the 
systems roach to training program development . 

' The reason people have avoided using ISD, f ™. ^t™^^ 
evaluator's point of view, was the ever present attitude of resisting 
change when the need for change has not been f^'*}***^- 
ledgl and understanding of the ISD system, and because full application 
of the ISD methods is time consuming and somewhat costly. 

Training Evaluation 

A definition of training evaluation was read to the training 
evaluation specialist respondents at the outset of the interview. 
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Training evaluation was defined "...as the determination of how well 
learning objectives were identified, the efficiency of the ISD process, 
and the proficiency of graduates in performing the jobs for which they 
were trained." A set of questions was then presented to the inter- 
viewees, relative to training evaluation. 

The first four items concerned the updating of courses to meet 
changing requirements. All respondents, but two, were involved with 
courses that had been periodically changed or modified. Most respond- 
ents indicated that the courses were modified, on the average, about 
once a year. However, it appears as though some courses were modified 
as frequently as at the end of each course, and as infrequently as 
every other year. A percentage breakdown of the responses indicated 
that in approximately 50 per cent of the cases courses were updated 
about once a year; in 20 percent of the cases there were revisions at 
the end of each course, and in another 20 percent courses were updated 
semi-annually; 5 per cent underwent revisions once every other year 
and the remaining 5 per cent could not be pinpointed. Course developers 
were usually informed that a course revision was needed as the result 
of information provided in student critiques, written reports (e.g., 
field evaluations, periodic course reviews), and conferences. 

Course Review Techniques 

The typical course review techniques, as separate from field 
evaluation, used were student critiques/student interviews, supervisory 
reviews, and reviews of field requirements. Less typical course review 
techniques mentioned were instructor reviews and course document reviews. 

The respondents were also asked to evaluate the course review tech- 
niques used for obtaining information that could be used for modifying 
a course. The results to this evaljation are presented in Table 6. 



Tabi • 6 

Evaluation of the Course Review Techniques Used 
CN = 21) 

Classification Per Cent of Respondents 

Superior 5 

Very good 32 

Good 29 

Fair 29 

Poor 5 
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A small percentage of the training evaluation respondents regarded 
the techSaueruscd in the courses with which they were most pillar 
2 Neither "Verier" or "poor." The majority of the responses tended 
S£4 Wd the middle of the scale. Approximately one- third of the 

indicated that the technics used in course reviews were 
'VerTgSd!" An equal percentage felt the techniques were "good" and 
again, about one-third thought they were "fair. 

Insights into the rationales behind this less than op timsn indi- 
cation welesou^t through the question, "What was the principal problem 
wUn^hTrourse^view technique" ployed?" Tlie single^t fr^tly 
iucntifiod problem was the subjectivity or biases ^twer^ allowed 
to creep into the evaluations. A close followup to ^ bl L°Ll 
iLtivitv was that some course reviewers were not skilled in the course 
review tecnnlquS em^yed. These comments suggest the need for training 
nrocra^ WMchwould [prepare course reviewers in the techniques of 
^revaluation including the use of objective techniques (such as 
S scales, checklists, etc.). Scy ^^fc.^^j^ 
of Technical Training Courses and Students i^iUl- ^-|^» Z^Jl 
by Applied Psychological Services as a sepa rate part of the current 
pUogS! my also help to provide at least a partial solution in this 
regard. 

Other problems mentioned as inherent in the course review tech- 
niques used at the training centers were: cumbersomeness of certain 
tracing evaluation forms and, in some cases, performance of course 
r^vi^lhf technical school personnel rather than by the training 
evaluators. The solutions to these problems are self-evident. 

Identification of Areas of Over and Undertraining 

The problem of over and undertraining is of concern to any training 
program that hS cost effectiveness as a goal. The training evaluators 
w^r1!^cco?dingly, asked which course review technique, as separate from 
field evaluations best identified areas of over and undertraining. 

training evaluators generally thought that while in -house course 
revieTtichniques might identify areas of undertraining, no course 
rlli™ telhniqul! Ssuch, could be used for identifying areas of over- 
SaiSne Rather, they suggested that an experimental program associated 
wi?h achievSnrtests be undertaken to determine areas of successful 
^rforma^e For example, if as a result of an achievement test, it was 
HsIS that JhHtudents were all performing satisfactorily in a 
particular subject matter area, then perhaps this area could also be 
Mrformed satisfactorily if slightly less time were spent on it during 
the course? The time saved could be devoted to other subject matter 
Sat ?he siudenls find more difficult to master, or the course could be 
conmensurably shortened. 



37 

ERIC . 4iJ 



tf9 



The training evaluators ilso indicated that the subjective comments 
contained in student critiques and instructor suggestions often identify 
areas of over (as well as under) training. laentny 



Specific Use of Course Reviews 



The interviewees were furtiier asked if they were aware of any 

™ lnC !- e ; nt in . vlhich a cour se review identified a problem and in 

^tSSrS % a £l°Z SfJ^S' 1716 ******** described the^oMem 
and statec how it was handled, who was responsible for instituting the 

C ^ tl0n : ^ J 0 ".* 1 * ^^^ation was beneficial, ^iimately 
"L£L C «?%£ f * he interviewees responded to this item? Inlbourthree- 
S he lnc d dent f related, the Training Evaluation DivisionTof 

?t£T:t 1Ve ^Sfi trainin S schools **« responsible for intro- 
ducing a change in either the course content or course administration 

^t™ 0 " 6 "^ 11 Cf inC j dents Wre ^ Sclo™ aS' 

£2EE£ r supervisors and, as before, either course content or course 

n^tn^ ^ ^ "?f aff f t€ *' For exam P le * ^ one incident the training 
evaluation department found certain lectures of a course to be dull and 
uninteresting. Students were interviewed, and this finding washer ified. 
^ tll l?^ fi Uat S rS ^sted to the instructors that the presentation 
dlcltd t^it m ^resting. The instructors 

™J e ? pl ? y the Progranmed instructional technique in place of 
the routine lectures. In addition to an increased interest level a 
time sfiv noc une ; n tw>* — a iuiwwi ieve± , a 



time savings was introduced. 
Field Evaluation 
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4 * evaluation, as separate from course reviews, was inquired 

SSilS 01 ^*- separate set of questions. The field evaluations usually 
involved critiques offered by both graduates of the course and thTsuner- 
visors of these graduates. Graduates usually critiques the training 
SnEE^* ^ TCh ool whereas the supervisors critiques the graduates' 

2- ? V*^ 0 ™ on the job. The respondents indicated a progrTof 
n^ tr ?i2 lng Performance testing to be a field evaluation technique in 
tfLll inst ^ ccs (e.g., three respondents indicated that posttraining 

^XTwiS! evaluation technique " ^ cour - °" -** 

The results of the field evaluations usually were fed back to the 
^fi^ n ? e yaluators through written reports or by way of the quest ion - 
Z JL S J hat CO f Se grates and supervisors completed. Another way 

K^^Lg^^ 5 wre - da of field evaluation ™ lts 

The adequacy of field evaluations, as a technique for obtaining in- 
formation that can be used for modifying courses, was evaluated by the 
training evaluators. Table 7 presents the result of this evaluation 
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Table 7 

[Valuation of the Field Evaluation Techniques Used 
' CN = 21) 

Classification Per Cent of Respondents 

Superior J£ 

Very good 48 

Good 33 

Fair ° 

Toor J 

No response 5 

Table 7 indicates that field evaluation techniques were never 
placed in the lowest categories, "fair" or "poor The interviews 
were obviouslv favorably disposed toward the field evaluation tech- 
SSeSTSS 62 per cent claiming they were 'Very good" or "superior" 
and 33 per cent indicating that they were 'good. 

There is a striking difference between this evaluation of the 
course review techniques and evaluation of the field evaluation tech- 
niques (cf. Tables 6 and 7). Whereas none of the respondents thought 
of the field evaluations as "fair" or "poor," 34 per cent thougt the 
course review techniques were classified as either "fair or 'poor 
Sixty -two per cent of the respondents classified the field evaluation 
techniques as "superior" or 'Very good," but only 37 per cent thought 
similarly of the course review techniques. 

When the interviewees were asked to state the P^PalfS^.. 
with the field evaluation techniques, the "careless" and "indifferent 
manner in which the questionnaires are completed was most frequently 
mentioned. The other problems mentioned were that snail samples were 
used and that the comments were very subjective. The respondents, in 
manv cases, claimed the superiority of the personal interview oyer 
the mail questionnaire because interviews allow fuller exploration of 
the basis for remarks. Personal interviews are, however, more time 
consuming and demand a skilled interviewer. Accordingly, the question- 
naire technique is often adopted as a matter of expediency. However, 
it is nossible that modification of presently employed questionnaires 
might produce more meaningful information. 

The interviewees indicated that evaluations produced by the 
craduates yielded the most meaningful and useful data- -both in regard 
to areas of overtraining and areas of undertraining. This does not 
implv that supervisory evaluations were not regarded as having _ value 
in this area, but rather that they provided less meaningful information 
than that provided by the graduates. 
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Approximately 60 per cent of the sample responded to the 
critical incident type item (item 17) which inquired into incidents in 
which field evaluation results were used by course developers. In 
all but three cases, the training e valuators were responsible for 
instituting the resultant change; instructors and headquarters 
personnel were responsible for modifying the course in the three 
cases. Tn every incident related, the course content was modified, 
and the modification was found reasonable and profitable. An example 
of a specific incident follows. As a result of a field evaluation, 
it was discovered that graduates of a course were not trained on one 
of the two computers used in the field. Training evaluators suggested 
that the course be updated to cover both computers. The instructors 
modified the course to include the computer that had been omitted. 

Student Measurement Specialists 

Thirty student measurement specialists were sampled at the 
five Air Force technical training schools visited. The interview with 
the student measurement specialists consisted of two major sections. 
One section dealt with the specifics of the student measurement tech- 
niques employed at the training centers; the other involved an 
evaluation of the ISO program. The IS) section of the interview will 
be discussed first. 



ISP Evaluation 

The student measurement specialists regarded the ISD program 
with moderate favorableness for achieving the objective of graduating 
students who meet job performance requirements efficiently and 
effectively. When asked to rate the ISD program on the five category 
scale which ranged from "superior" to "poor," 10 per cent of the 
student measurement specialists placed the program in the bottom two 
categories, "fair" and "poor." Forty-seven per cent considered the 
ISD program as "superior" or "very good," and 33 per cent considered 
the program as "good." The results to this evaluation are presented 
in Table 8. 
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Table 8 



Results of Lvaluation of the ISD Program 
hy StiKlent teasurement Specialists (N « 30) 



Classification 



Per Cent of Respondents 



.Superior 
Very good 



Good 
Fair 
Poor 



No response 



20 
27 
33 
7 
3 
10 



When later in the interview the respondents were asked to 
evaluate the ISD program on the quantitative scale which ranged from 
0 to 100, the mean obtained value was 62 with a standard deviation of 
22. This value may be compared, for reliability purposes, with an 
expected value (on the basis of the categorical data) of 55.8. This 
value (56) is again lower than that assigned by the ISD specialists 
(78) and is identical to that assigned by the training evaluation 
specialists. 

The problems most frequently mentioned by the student measure- 
ment specialists in regard to the ISD model were: insufficient number 
of trained 19) specialists, inadequate general information program on 
the advantages of the ISD technique, and difficulty with certain of 
the ISD steps. 

Again step 1 (analyze system requirements) was said to cause 
difficulty, and in some cases task analyses had not been performed 
for courses that were otherwise modified in accordance with the ISD 
model . 

The student measurement specialists suggested several solutions 
to the problems that were experienced in applying the ISD method. 
Most frequently mentioned for improving the use and application of 
ISD was, again, the development of more and better trained ISD 
specialists. Other suggestions for improving the use and application 
of ISD were: (1) increase the flow of information in regard to the 
ISD program so that the training personnel responsible for developing 
courses on the basis of the ISD method would meet with less opposition 
and with greater cooperation, (2) allow more time to prepare a course 
when using the ISD method, and (3) reduce the volume of required 
paperwork . 
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Tlie need for an Lie reuse in the tijne allowed to prepare a 
course by using the IS!) may be a result of the fact that the technique 
lias often been new to the personnel .assigned to course development. 
As IS!) specialists and others assigned to developing courses gain ex- 
perience with the ISD method, it is likely that less tijne will be 
required to develop courses with the procedure. 

There seemed to be agreement among the student measurement 
specialists on the reason that the ISD method may sometimes be avoided. 
The student measurement specialists, very much like the other specialty 
groups, thought the ISD method should be avoided because course 
development personnel were resistant to change and did not understand 
the method. Moreover, they claimed that use of the ISD method 
involves too much work. Some interviewees were of the opinion that 
if more I Si) specialists were available, the amount of work required 
from any one person in developing a course would be diminished, com- 
mensurate with the increased number of ISD specialists. 

Student Measurement 

Prior to questioning the student measurement specialists in the 
area of student measurement techniques, a definition oJ student 
measurement was read, 'ihe definition stated, "We define student 
measurement as measures for the purpose of evaluating individual 
student achievement and progress." Table 9 presents the results 
to the first question in this area. This question asked the inter- 
viewees to identify the different student measurement techniques 
that were employed in the course they identified at the beginning of 
the Interview as being the one with which they were most familiar. 



Table 9 



Student Measurement Techniques Employed 

(M = 30) 

Technique fluiiiber of Responses 



laid of block written test 28 

Performance test 26 

Criterion checklist 18 

Instructor rating 13 

Programmed instruction test 7 

Oral test 7 
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Table 9 indicates that the end of the block written test and the 
performance test are the most frequently employed student achieve- 
ment measurement teciuiiques employed. The criterion checklist, 
which is gaining in popularity, was identified as a technique used by 
somewhat fewer student measurement specialists. Instructor ratings 
were used, but less frequently than the prior three techniques, and the 
programmed instruction test and the oral, test were relatively infre- 
quently used. 

End of Block Written Tests 

A set of subsequent items inquired specifically into end of 
block written tests. Absolute scores (as opposed to criterion refer- 
encing or norm referencing) were used to determine the passing 
point in all cases. Item analyses, for the purpose of appraising the 
usefulness and effectiveness of each item, were conducted in all 
courses where end of block written tests were used. They were typical 
item analyses in which difficulty indexes are obtained. In only a 
few cases, the respondents indicated, were the end of block written 
test results correlated with other measures of performance to deter- 
mine the degree of relationship between two different measures of 
performance. In the few cases where these correlations were obtained, 
the end of block written tests were correlated against performance 
test scores. 

Although many respondents thought that end of block written tests 
were effective student evaluation devices, many also thought that 
they were ineffective tools. A small number of the respondents 
reported that end of block written tests had motivational value (i.e., 
students faced with written tests will study for them). These tests 
were also lie Id to be comprehensive, practical, and objective (i.e., 
the short answer type , rather than the essay type) . 

In this set of questions, the respondents were also asked to 
identify problems involved in the use and development of these tests. 
The major problem identified was test construction difficulty (e.g., 
writing effective items). Students were always informed of their end 
of block written test results. Those who did not achieve satisfactory 
results usually were required to repeat the block. In some cases, 
special assistance was provided to the student who failed a written 
test. The student was then retested. If failure still resulted, the 
instructor could recommend that the student be dropped from the course. 

Performance Tests 

Performance tests were investigated in a manner similar to that 
in which tlie end of block written tests were investigated. Perform- 
ance tests (sometimes referred to as work sample tests) were eimployed 
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in all but four of the courses involved. In some cases, absolute 
scores were used to separate passing and failing students, but in 
almost as many cases passing or failing was norm referenced. The 
respondents felt that tlie value of performance tests rested in their 
practicality and their use as a tool for keeping track of the progress 
of a student. Practical, in this context, lias several meanings. It 
could mean that actual equipment, i.e., the same equipment that is used 
in the field was used in the test, or that the tasks and/or problems 
were closely related to those found in the field, or that if skills 
were taught in the course the performance test was a valid instrument 
for measuring achievement. 

The most frequently mentioned problems with the performance 
tests were of a test administrative nature. Unlike a written test, 
the performance test allows only one student to be tested at a time. 
Thus, unless the classes are small or there are several test adminis- 
trators, the performance test has to be of limited duration; otherwise 
testing student achievement can require considerable time. Three other 
performance test related problems were mentioned with some frequency: 
equipment for performance testing is sometimes difficult to acquire, 
the lag between equipment requisition and equipment receipt is too 
long, and performance test scoring is often subjective in nature. In 
regard to equipment acquisition problems, it seems that inexpensive 
mockups and/or paper-and-pencil performance tests might offer some 
solution. 

As with the end of block written tests, students were informed 
of their performance test results. Those that did not receive satis- 
factory results received additional training or special assistance 
and/or were "washed back." Only in very few cases (where the perform- 
ance was exceptionally poor) were failing students recommended for 
elimination from a course on the basis of the performance test results. 

Criterion Checklists 

The subsequent set of items involved the application of the cri- 
terion checklist. Two-thirds of the student measurement respondents 
were familiar with courses in which the criterion checklist was employed 
as a student measurement technique. In most cases, satisfactory and 
unsatisfactory performance was determined on a criterion referenced 
basis, tiowever, some respondents reported that satisfactory-unsatis- 
factory performance was determined by subjective evaluation, and a 
small number of respondents replied that the criterion checklists 
they used had numerical scores given for various aspects of performance 
and an absolute score was used to determine a student's final score. 
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The criterion checklist was almost universally said to represent 
a thoraU practical, and valuable teduiique for keeping track of 
rtJ3SSpiSg?S5. In some courses, the criterion J^cklist was used 
as a precSfor to the end of block written test Here, student shau^ 
7l ^rform satisfactorily the items on the criterion checklist netore 
£ fouKfdTS test. Thus, the checklist bec^e a «adung 
aid as well as an achievement measuring instrument. As witft tne 
-rfo^nce Sst, the criterion checklist was seen as. practical because 
?f,rstu^nts were called on to perform as they would in the real 
sUuation Alio! Se student was evaluated by an instructor who, in 
thlf situation! could function in much the way that a recent graduate's 
supervisor would function. 

The problems inherent in the criterion checklist fell into the 
follow/categories: administrative, scoring subjectivity, costly, 
£d SSfcSt £ construct. For example some "SgT^taT^ 
that they had to develop the criterion checkl is t bef ore 
knowledge of the system requirements (step 1) . This res ^ eo "\^i 
difficulty in developing criterion objectives. Perhaps this problem 
is rented To She so|Sftication of the test developer, his knowledge 
of the subject matter, and the needs of the field. 

Some of the same problems inherent in. performance test development 
and a Snistration were also involved in the criterion checklist. In 
^arrtrtelfatoTislrator reliability, some interviewees suggested 
^^vision of the e valuator while he completes the checklist or 
o^SSng^valuaUcS from two independent evaluators. Other solutions 
Tc ?his orobleTmieht be sought through training programs in which 
p°ro^L P ote^or b!as?^ P ro^her to evaluating performance, and 
completing the checklist are fully explored. 

As with end of block written tests and performance tests, students 
were flwSs informed of their criterion checklist results. Those who 
Td not^eve Satisfactory results were either recycled or received 
fd3i?ional training or special assistance. They then repeated^ 
test they had failed. Only in rare circumstances did an interviewee 
^ort tL?1 student, on the basis of criterion checklist results, 
was incapable of completing the course satisfactorily. But, if 
n^essar^; students were dropped from the course on the basis of 
criterion checklist results. 

'flie retest procedure may suggest the need for parallel tests 
rather than repetition of the same test. 
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In summary, the perceived advantages and disadvantages of the 
three student achievement measurement techniques, as seen by student 
measurement specialists, are presented below: 



Test 



Advantages 



Disadvantages 



Written 



Performance 



1. 


keeps track of student 


1. 




progress 


2. 


1 


objective 




3. 


motivating 




4. 


thorough 




1. 


keeps track of student 


1. 




progress 




'7 

*• ■ 


pi..ctical 


2. 






3. 



Criterion 
Checklist 



1. keeps track of student 
progress 

2. practical 

3. thorough 



ability 



individually administered 
subjective scoring 
difficult to obtain 
requisite equipment 

1. too long to complete if 
individually administered 
by one test administrator 

2. subjective scoring 

3. difficult to construct 



Needed Student ffeasurement Techniques 

Item 29 of the interview with student measurement personnel 
queried into student measurement techniques which are not currently 
used but which the student measurement specialists would like to see 
used. The results indicated that the testing devices currently beine 
used were considered to be satisfactory. This may suggest some lack 
of knowledge of more advanced testing practices such as confidence 
like computer assisted testing, hierarchical testing and 

Feedback to Course Developers 

Test results were usually provided to course developers by way 
of student records, written reports of various kinds, school records 
word of mouth, and in some instances at meetings. About 20 per cent 
of the interviewees reported instances in which the course developers 
had instituted corrective action on the basis of test results. The test 
results were used by the course developers in several different ways. 
If course changes or revisions were instituted as a result of student 
evaluations, the change was in the form of modifications to existing 
examinations or equally often in the form of changes in instructional 
techniques. On a less frequent basis, course outlines were revised or 
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™ derived ^f^ilieVrfornian^ rcquiivracnts. On a very infrequent 
lSu/^r^luii« wen, used by course J^JW» &£j£r 
the selection criteria for the course and to evaluate the "^ructor. 
This latter employment may also represent a misuse oi test results 
l^ause sud» ukc encourages the instructor to teach the test. 

An example of a situation in which a modification was made as a 
result of text findings involves an item analysis of a written test. 
S founulhat ,nany of the high. scor ^tujtatt ^ «J»^ 
reticular item on the test. Hie instructor and instructor supervisor 
Seciued to record the item. The item demonstrated proved discrim- 
inability in subsequent test administrations. 

Review and Revision of Student Measure ment Techniques 

When inquiry into the frequency with which student measurement 
toclmtaL were reviewed and modified as a result of the reviews was 
^T^^er Snt ofthe respondents indicated that student measure- 
mont tcchniaucs were reviewed at the end of each course, 25 per cent 
SUd KiT reviewed "aperiodically," 13 per cent said yearly, 
six %v c% SSaS Sat they i*» reviewed quarterly, and another 
six pr cent said »L. .uannually." Evidently, firmer ground rules are 
needed here. 

Modifications to measurement techniques, resulting from these 
reviews occurred as frequently as at the end of each course for 17 
Tr Snt oi tlTin^rviewees aiid semiannually for another 17 per cent. 
Twcn^hree *r centTf the respondents could not fix a frequency for 
Ite fictions that resulted from student measurement reviews, but 
ndic^ed t SSTSdiflcaSons did result from the reviews Seven per 
cent claimed that modifications to the tests were made about every 
quarter and anther 7 per cent claimed that they were made . yearly. 
Fiirillv 3 ner cent of the interviewees replied tUt modifications we.e 
^fe about o^ce every other year. The remaining 26 per cent of the 
hnervie^s Sid not respond because they had insufficient information 
™ m mrttcr Reviews of student measurement tecliniques can be 
^^ ^ttTS^ t^t they yield modifications (if required) 
Etta inurement devices. On the other hand, it seems that modifi- 
cation on the basis of the results from one class might represent an 
inefficient sample for firm conclusions. It would seem that the 17 
^^V^^i^U who modify tests semiannually are per forming 
thl< upda?c at an acceptable frequency; others may be performing the 
update too frequently or not frequently enough. 
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Student Critiques 

Student critiques were completed in all of the courses 
discussed by tlie interviewees. The critiques were reviewed by many 
different persons, sections and departments at the schools, from 
the instructors up to the base commanders. About one-third of the 
interviewees related an incident in which the course developers 
took corrective action on tlie basis of the critique forms completed 
by the students. In these incidents, the course supervisors were most 
often identified as being responsible for the corrective action. 
However, one incident was related in which the instructor was respon- 
sible for the change and two incidents involved other departments at 
the school. The incidents were varied so that the resulting correc- 
tions were applied to the course content, tests, and physical facili- 
ties. For example, in one instance, a student complained that the 
lighting in the laboratory was inadequate. The tasks performed in 
the laboratory required a high degree of precision, i.e., soldering 
connections. The lighting in the laboratory was measured and a work 
order requesting an increase in the lighting was submitted. The 
lighting level was increased, with the result that this unfavorable 
condition is no longer a negative factor affecting the learning 
environment. 



Training Managers/ Instructors 

The interview for training managers/ instructors consisted of 
three sets of items: ISD, training evaluation, and student measure- 
ment. 

ISP Evaluation 

Initially, the interviewees were requested to evaluate tlie ISD 
program for achieving its stated objectives of graduating students 
who meet job performance requirements effectively and efficiently. 
The results to the item are presented in Table 10. 



48 

53 



•Li; 



Tabic 10 

Results of Evaluation of ISD Program 

bv Training Managers/ Instructors 
CX * 52) 

Classification Pe r Cent of Respondents 

Superior 10 

Very good 44 

Good 23 

Fair • 6 

Poor 0 

Nio response 17 



Seventeen per cent cf the respondents were realtively unfamil- 
iar with the ISD program and therefore could not evaluate the program. 
Of the remaining 83 per cent, 54 per cent placed the program in the 
"very good"/"superior" classification, 23 per cent classified the 
program in the middle category "good," and 6 per cent considered the 
I Si) program as "fair." None of the respondents considered the ISD 
method to be "poor." 

As for the other groups interviewed, the general evaluation of 
ISD was also approached through the magnitude estimation technique 
in which the respondents were asked to evaluate ISD on a scale which 
ranged from 0 to 100. This evaluation yielded a mean value of 69 
with a standard deviation of 18. Obviously the training managers/ 
instructors regard the ISD favorably. (The anticipated mean value 
for the quantitative scale, on the basis of the categorical scale, 
is 53. Here, there is a greater difference between the anticipated 
and the obtained than for the other training specialties interviewed. 
This suggests greater response unreliability for this interviewee 
group than for the other groups). This value (69) places the reactions 
of the training managers/instructors to be lower than those of the 
ISD specialists but above those of the student measurement and the 
training evaluation specialists. 

Typical problems (in order of frequency of response) reported 
by the training managers/instructors in developing new courses using 
the ISD Model included: difficulty in meeting the requirements of the 
ISD steps, inadequate understanding of ISD steps, lack of information, 
lack of personnel, insufficient time to prepare courses, inflexibility 
of system, and lack of cooperation from instructors. 
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The suggestions for improving the ISD program offered by the 
training managers/instructors were very much, like those of the other 
specialties interviewed. Among the suggestions were: more and better 
trained ISD specialists, increased acceptance of the program at the 
training centers, more complete job analysis by going into the field 
instead of using mail questionnaires, assignment of subject matter 
specialists to work with ISD specialists (this will tend to speed up 
the process of preparing a course because it will relieve the ISD 
specialist from taking the time to learn the subject matter), improving 
the administrative aspects of ISD by reducing the amount of paperwork 
required, and raising the screening and selection criteria so that 
tne Air Force will have a superior corps of ISD specialists. 

The training managers/instructors maintained that ISD is avoided 
because it is time consuming, because its objectives are not under- 
stood, and because application personnel are resistant to change. 
Most training manager/ instructors supported the employment of ISD for 
such reasons as: "it presents an efficient means of training people 
to meet job performance requirements," "less slipshod and therefore 
better courses are developed," and "the graduates are more appropriately 
trained." On the other hand, several inten "ewees indicated that their 
support for the ISD program was based solely on the fact that they were 
ordered to use it. 

Best Aspects of ISD Method 

During one aspect of the interview, tlie training manager/instructor 
was presented with an adjectival list of 10 words, all positively 
keyed, from which they were asked to select the three adjectives they 
thought best described the ISD procedures. Table 11 presents the list 
of adjectives, the number of times each adjective was selected, and 
the resultant hierarchical rank order. 

Table 11 

Positive Adjectives Selected to Describe the ISD Model 



Adjectives 


N_ 


Rank Order 


Organized 


29 


1 


Effective 


27 


2 


Efficient 


14 


3.5 


Profitable 


14 


3.5 


Complete 


13 


5 


Appropriate 


11 


6.5 


•Ielpful 


11 


6.5 


Good 


5 


8 


Dynamic 


4 


9 




50 



55 



Tabic 11 indicates that the four most frequently selected adjec- 
tives were: "organized," "effective," "efficient," and "profitable." 
The adjectives appearing at the bottom of Table 11 were not necessarily 
thought of as inappropriate, but rather as less appropriate than those 
at the top of the list. Quite obviously, the possibility exists that 
the provided list was incomplete and that the respondents might have 
misunderstood the meaning of any one or more of the adjectives. 

This item was followed by one which requested that the inter- 
viewees state, in their opinion, the three best aspects of the ISD 
program and to rank order the aspects from 1 to 3 (the rank order of 
1 was the highest; 3 the lowest). 

The responses were categorizable in six separate categories. 
The frequency with which each category received a ranking of 1, 2, or 
3 was tabulated, and a total score (rank order) for each category 
was calculated. Table 12 presents these data. 

Table 12 



Rank Order of Best Aspects of the ISD Model 



Category 




Frequency* 




Score 




1 




3 




Systematic, standardized, organized, 
logical, validates instruction 


17 


16 


13 


96 


^torc appropriately prepared 
graduates 


10 


5 


6 


46 


Job oriented 


8 


7 


* 


41 


Identifies course objectives 


3 


10 


5 


34 


Improved tests and measurements 


3 


3 


6 


21 


Cost effective 


2 


1 


4 


12 



* Frequency of a ranking of 1, 2, or 3. The total N for each 
column varies as a function of the number of responses. -.Some 
interviewees identified only one or two of the ISD model's 
best aspects. 
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In order to obtain the scores (Table 12) a weighting scheme was 
used sudi that a rank order of 1 was weighted with a 3, the rank order 
2 was given the weight of 2, and the rank order 3 was weighted with a 
1. The weights were used as multipliers and the three products were 
added together to arrive at the final score for the category. Thus, 
for the first category listed in Table 12, the score of 96 was 
obtained as follows: 17(3) + 16(2) + 13(1) - 96. 

The first category, the category with the highest score, per- 
tains to the methodology inherent in the ISD model. Therefore, the 
training managers/instructors were most favorably impressed with the 
ISD methods. The least impressive aspect (but nevertheless among 
those identified as the "best") was cost/effectiveness. The results r,f 
this analysis are interesting from the point of view that the respond- 
ents, as training managers/instructors, should be concerned more with 
the methodology of a course development system than with some other 
aspect of the approach. 

Worst Aspects of ISD Method 

In a parallel manner, the interviewees were queried on the neg- 
ative aspects of the ISD method. Initially, they were ask^d to select 
the three worst aspects of the ISD model from a list of ten adjectives. 
Tiie results are presented in Table 13. 



Table 13 

Negative Adjectives Selected to Describe the ISD Model 



Adjectives N Rank Order 

Tedious 29 1 

Costly 26 2.5 

Limited 26 2.S 

Static 6 4 

Mundane 3 5 

Unsuitable 2 7.5 

Unworkable 2 7.5 

Unsuccessful 2 7.5 

Weak 2 7.5 

Poor 1 10 
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Within the listed adjectives presented, the three worst aspects 
of ISD were that it is "tedious," "costly," and "limited." The other 
adjectives, although negative in connotation, were hardly selected by 
the respondents. The choices of "tedious" and "costly" are compatible 
with the responses of other groups to various aspects of the interview, 
as well as with the responses of the training managers/ instructors to 
other questions . The choice of "limited" suggests that, to some extent, 
the respondents believed the ISD teduuque not applicable to all courses 

The training manager/ instructor respondents were also asked to 
identify the three worst aspects of the ISD model, using their own words 
and to rank order the aspects. The ranking of 1 was equal to the worst 
and 3 was equivalent to the least bad. The data were treated in the 
same manner as was used for best aspects, described above. The results 
arc presented in Table 14. 

Table 14 

Rank Order of Worst Aspects of the ISD Model 
Category Frequency* ^ Score 

Developing curricula, objectives, 
™a 9 8 10 53 

9 6 8 47 



and tests 

Costly and time consuming 



Limited to teaching skills, not 
enough student -instructor 
interaction 



10 5 2 42 



Instructors and personnel inade- 
quately trained in ISD methods 6 6 2 32 

Analyzing system requirements 4 4 2 22 

Feedback from field is slow 12 1 8 

* The total N for each column varies as a function of the number 

of responses. Some interviewees identified only one or two of 
the ISD model's worst aspects. 
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As suggested in the responses, the most faulty category of ISD, 
according to training managers/instructors, was tJiat skill and exper- 
ience is needed to develop a course curriculum, objectives and tests by 
the ISD standards. Although the feedback that was built in to the 
model did not work to the satisfaction of the TM/Is, it was on the 
bottom of the list. 

Evaluation of ISD Program 

One interview question requested information on the procedures, 
if any, that were instituted for evaluating the ISD program. Most of 
the respondents were unaware of any such procedure. However, several 
indicated that certain formal procedures were used, i.e., cost 
comparisons and comparison's of graduate proficiency between ISD 
trained graduates and others. Evidently, increased emphasis is 
needed in this regard. 

Course Review 

The opinions of the training managers/instructors in the course 
review sphere were sought in a separate set of questions. All but three 
of tlie interviewees said that courses were periodically updated to 
meet changing requirements. Most courses were so modified approx- 
imately once a year, once every six months, or at the end of each course. 
The information indicating that a revision was necessary usually came 
in the form of feedback from the field, changes in regulations and 
manuals, suggestions from instructors, and student critiques. The 
information was fed back to the training managers/instructors through : 
(1) written reports such as graduate survey questionnaires and field 
supervisor questionnaires, (2) meetings and conferences, and (3) word 
of mouth, as might come about through interactions with other 
instructors and students. 

The course review techniques, (as separate from field evaluations) 
that were mentioned most frequently were student critiques, suggestions 
from instructors, reviews of field requirements, and supervisor reviews. 
Opinions of course review techniques, as evoked by a catagorical scale, 
are presented in Table 15. 
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Table 15 

Evaluation of the Course Review Techniques Used 

Classificatio n » of Respondents 

Superior W 

Very good 5^ 

Good *3 

Fair 6 

Poor * 

No response ' 

\ very small percentage of the training managers/instructors 
were dissatisfied with the course review techniques that were used. 
Cnlv eight per cent rated the course review techniques as fair or 
^rt" Thf latest proportion of respondents indicated satisfaction 
("good," "very good," or "superior") with the techniques. 

The principal course review technique problem, as seen by these 
respondents, wJ that the reviewers were subjective (as might occur 
with student critiques and with some instructor suggestions) and in 
Sny insmces untrained and/or ill prepared, to review and evaluate 
the r nurse. An example of a course review situation being performed 
by a IS ihaTfun/prepared training evaluator mis provided by an 
interviewee. In this cake, the review wa> performed by Mr Tracing 
S personnel who were not fully aware of specific local Problems. 
The result, according to the interviewee, was a set of suggestions 
which was unrealistic. 

Mother course review technique problem mentioned often in the 
interviews with the training manager/ instructors was the problem of 
lag time. It seems that the time between the reviews and the imple- 
LntaSon of the recommendations for ^vision was too area tto be of 
maximum benefit. Additionally, the training ^Sers/mstructors 
sometimes thought that instructors are overburdened when given the 
aSuonal^sp^nsibility of reviewing course mater al, new texts, 
otr for the purpose of updating a course. These interviewees 
fet tliS c^rreviewTshould be performed by training evaluation 
specialists and not instructors. 

lite training managers/instructors most often indicated that the 
problem of overtraining can best be identified by student critiques 
aTiStruc?or suggestions. The problem of underlining, on the other 
Snd^ght be identified best by the same two sources in addition to 
supervisor reports from the field. 



A critical incident item was used to obtain insights into course 
review results that were actually used and into the mechanism for 
such implementation. Hie interviewees were asked to identify a course 
review result, the needed revision, who was primarily responsible 
for bringing about tlie change, the outcome, and an explanation of 
how the results were helpful. Almost 40 per cent of the interviewees 
responded to tlus item. In every case related, the instructors were 
responsible for recommending the course revision; in one case a change 
was introduced to an existing test, in another to course equipment, 
and in the remainder 'o course content; in every instance the modifica- 
tion was helpful and beneficial to those concerned. An example of a 
reported incident follows. The students complained, in the student 
critiques, that tiiey were having difficulty learning a particular 
subject nutter area. The techniques used in presenting the material 
to the students were a lecture and slides. The instructors replaced 
tJie lecture and slides with a motion picture, in which a step by step 
procedure was demonstrated. Test results indicated that the problem 
was solved. 



Field Evaluation 

iiighty per cent of the respondents claimed that field evaluations 
were conducted for the purpose of updating the course. For most of 
the courses involved, the field evaluations were conducted yearly. 
For some courses, they were conducted semiannually, and for other 
ccurses they were conducted quarterly and every other year. The field 
evaluation techniques consisted of supervisor evaluations of graduates 
and graduate critiques of the training they received. The field 
evaluation results were fed back to the training manage rs/instruc tors 
via written reports, meetings, and word of mouth. 

As indicated in Table 16, fifty- two per cent of the training manager/ 
instructor sample considered the course review techniques as "superior" 
or "very good," wlicreas twenty-seven per cent considered the field 
evaluations as such. Similarly, 32 per cent considered the field 
evaluations to be "fair" or "poor," whereas 8 per cent thought the 
course review tediniques were "fair" or "poor" for obtaining information 
that can be employed for course update. 
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Table 16 




Classification 



% of Respondents 



No response 



Superior 
Very good 



Good 
Fair 
Poor 



8 

19 
17 
17 
15 
23 



The explanations for the relatively low evaluation were found in 
the enumeration of the problems with the techniques. The field eval- 
uation problems most frequently mentioned were: subjectivity, careless 
completion of questionnaires, insufficient sanples of field units, and 
inexperienced evaluators and interviewers conducting field visits. 
The interviewees suggested that: (1) respondents to the field evaluation 
questionnaires often comment in areas in which they are unqualified, 
(2) the field evaluations were not extensive enough to provide valid 
assessments of graduate performance, (3) field visits should be conducted 
more frequently and cover all conmands, (4) interview teams rather than 
questionnaires be used because more intensive investigation could be 
made through this technique, and (5) interviewers that go into the field 
be trained both in interviewing techniques and in the content area 
under consideration. 

Graduate questionnaires were considered by the training managers/ 
instructors, as for the other groups interviewed, to represent the best 
technique for determining areas of over and undert raining. Supervisor 
evaluations were regarded as a good source for identifying areas of under- 
training, but not overtraining. Fifty per cent of the sample of training 
managers/instructors responded to item 28, a critical incident item, 
which asked the interviewee to relate a specific incident in which a 
field evaluation result was used by course developers. In all but two 
of the incidents, so related, an instructor or instructor supervisor 
implemented the recommended modification. All the incidents, but one, 
involved modifications to course content. The one incident that did 
not involve course content was one in which a modification to the 
existing physical facilities was required. In all cases, the modifi- 
cations were successful. 
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In one instance the curriculum section was responsible for 
instituting the change. In this case, a new procedure was being used 
in the field for loading pallet trains. Since this procedure was not 
taught in the course, the graduates learned the procedire on the job. 
As a result of field interviews, the curriculum depart acnt learned of 
tlx? need for updating the course. The request was sub njtted to the 
course instructors, who revised the course so that the new procedure 
was included. 

Several of the questions that were included in the interviews 
with training managers/ instructors were also included in the training 
evaluator's interview. Cross comparisons of the responses o r the 
two groups indicates some opinion differences. For example, a com- 
parison of Tables 6 and 15, which presented the results to the evalua- 
tion of tiie course review techniques that were used in their courses, 
indicates the responses of the training evaluation specialists to be 
less favorable to the course review techniques than the responses of 
the training managers/ ins true tors. About one -third, in cadi sample, 
rated the techniques "good." However, while 52 per cent of the training 
managers/instructors placed the techniques in the "superior" or "very 
good" categories, only 37 per cent of the training e valuators did so. 
Similarly, while 8 per cent of tiie training managers/instructors rated 
the techniques as "fair" or "poor," 34 per cent of the training e valua- 
tors did the same. Since instructors play a more active role in the 
actual operations of reviewing a course (they were identified as an 
integral pari of a course review technique, namely, instructor sugges- 
tions), it is possible that they may be predisposed to view them more 
favorably. On the other liand this experience with the techniques may 
place them in an excellent position for providing valid evaluation 
state-ncnts. If merit is as smncd for the remarks of the training 
evaluators, then some modification in course review techniques my be 
indicated. 

A comparison of the assessments of the field evaluation techniques 
provided by the training manager/instructor and the training evaluation 
specialties also suggests opinion differences. (Tables 7 and 16). 
Sixty-two per .;ent of the training evaluators considered the field 
evaluation techniques to be "superior" or "very good." However, only 
27 per cent of the training managers/instructors evaluated the techniques 
as "superior" or "very good." None of the training evaluators placed 
the field evaluation techniques in the fair/poor categorization; 32 per 
cent of the training managers/ instructors considered the field evalua- 
tions, as used for updating courses, as "fair" or "poor." 



ERIC 



The inconsistency across the two different specialties suggests 
either tJiat each has met with greater success with one teduiique than 
with tlve other or that oik group possesses insights (or vested interests) 
not possessed hv the ether. If merit is assumed for the remarks of the 
training managers/instructors, then some modification in certain of the 
field evaluation techniques may be indicated. 

Student Measurement 

The various student measurement techniques investigated were: 
end of block (and unit) written tests, performance tests, criterion 
checklists, oral tests, instructor ratings, and programmed instruction 
tests. Passing and failing scores on end of block (and unit) written 
tests were based on an absolute score. The training managers/ instructors 
thought that the end of block written tests were effective for evaluating 
students for the same reasons given by the other training specialist 
groups interviewed. Although the end of block written tests were viewed 
favorably bv many of the interviewees, nevertheless several interviewees 
(quite correctly) did not think tiiat they are effective for evaluating 
students who are learning skills (as opposed to learning conceptual 
material). 

Some of the reasons given for the judgment that the written tests 
were effective were that: (1) the tests include one item each for every 
enabling and criterion abjective, (2) statistical approaches to test 
development are used, (3) the tests are periodically reviewed for the 
purpose of updating, and (4) the tests provide objective evaluations of 
student achievement. Written tests were said to be practical because: 
(1) students are familiar with their traditional format, and (2) they 
isolate the students strong and weak areas. 

Policy states that students receive the results of their tests. 
Students who did not perform satisfactorily on tests either received 
additional training and/or special assistance and then were retested or 
repeated the block and then retested. Although students could be 
dropped from the course (and were in some cases), this was not a popular 
approach to the problem and was reserved for very special circumstances. 
Those who were dropped from a course usually had problems that affected 
their progress in the course (such as physical or emotional problems) 
but which were not directly related to learning ability. 

Hie training managers/ instructors were asked if they were aware 
of a specific incident in which course developers took corrective 
action on tlx? basis of student measurement results. Approximately 35 
per cent of the interviewees responded. The instructor or course 
supervisor was mentioned as being responsible for fostering the use of 
the resulting recommendation. Modifications were made to course content, 
the equipment, or the tests. In all the incidents related the modifica- 
tions were found to be beneficial. 
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Student Critique 

According to the training managers/instructors, graduates com- 
pleted cretique forms which were reviewed by the instructors, brandi 
level personnel, department level personnel, and occasionally the 
instruction and measurement section, curriculum branch, technical 
school , and center commander. 

Sixty-one per cent of the training manager/ ins true tor interviewees 
related critical incidents in which corrective action was taken on the 
basis of student suggestions. Instructors, course supervisors, and 
curriculum section personnel were identified as those who implemented 
the corrective action. In the incidents related, action was taken 
in regard to course content, tests, equipment, and course administra- 
tion, ilie corrective actions were always found to be reasonable and 
beneficial. 

Evaluation of Instructional liffectiveness 

According to tlie training managers/instructors the criteria that 
were used to judge instructional effectiveness are the technical 
competence of students as displayed on their course tests, teaching 
ability as reviewed by course supervisors, the qualifications of the 
instructors, adequacy of textual materials, and adequacy and appro- 
priateness of equipment and facilities. The criteria used for 
judging instructional effectiveness generally did not differ for 
different courses. Not all the criteria listed above were used in 
judging instructional effectiveness in every course. 



lie fore the interview was terminated, each interviewee was given 
the opportunity to state anything lie thought would be of some im- 
portance to this field survey. The open-ended question, "Is there 
anything else you would like to tell me?" was asked to obtain this 
information. The following statements were typical for each of the 
job specialties sampled and represent current thoughts and attitudes 
within the various specialties. 



s 
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Instructional System Development Specialists 

The I Sll method is a superior training system and achieves 
better training than any otiier system 

ISD is good for trade ami skill courses but not necessary 
for tlie academic and knowledge courses 

Tliere is too much emphasis on cost effectiveness 

Increase the number of trained ISD specialists 

People at the centers blame the ISD method for the exces- 
sive time it takes them to develop a course- -the delays are 
due to their un familiarity, not the method 

If ISD specialists arc not provided with subject matter 
specialists when developing a course, then provisions should 
be made for tiiein to take the course 



Training Evaluation Specialists 

Training evaluation division often is assigned specialists 
who arc not qualified for the position- -they often come up 
through the instructional ranks 

Cooperation among the departments is less than good- -it is 
sometimes difficult for course managers to accept the results 
of training evaluations 

Restrict the training evaluation specialists' activities and 
rcspomtihilit ics to training evaluation and do not include 
other aspects of training 

Student Measurement Specialists 

Lxplain ISO objectives to course personnel --this could result 
in time and cost savings in developing courses 

The ungraded criterion checklist represents a great improve- 
ment --cncckl ists are more appropriate for skill training 

Increase standards for graduation from a course --low level 
graduates can return to the course as instructors and too 
often they are only mediocre instructors 

Preparing courses by the ISD method involves too much work and 
this has the ultimate effect of reducing instructional effect 

iveness 
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Training Managers/ Inst rue tors 



Too much emphasis is placed on student critiques 

Criterion check lists arc the best tiling to come along in testing 

I Si) is good for low level courses 

Standardize on the equipment used in tlic field so that retraining 
teclinicians would not he required so frequently 

Field evaluations are poorly conducted- -comments on the course 
content by people who are not recent graduates may not be valid 
and recent graduates are not sufficiently knowlcdgablc about 
their jobs to comment on the training they received to prepare 
them for the job 



Iiiiportancc of LSI) Steps 

The five LSI) steps were evaluated by each group of respondents 
for the purpose of determining which steps were considered to be 
most instrumental in achieving an effective instructional program. 
The rank ordering method was used to obtain this information. In 
order to avoid positional biases, the five steps were presented in 
their sequential order (i.e., steps 1, 2, 3, 4, 5) to one portion of 
the sample and in a randomly based disarray for another portion of 
the sample (i.e., steps 4, 2, 5, 1, 3). 

Hjc instructions to the interviewees were: 



Please assign rank ordcii, from 1 to 5, in this column 
so that we can determine which ISD steps you think help 
most to achieve an effective instructional program. The 
number one is the best, number 5 represents tiie worst. 
If you want to tie two steps, then assign both the average 
value between tlie two positions and skip two numbers. 
For example, if you wanted to rank ISD step 1 and 4 as 
tied for third position, you would assign the rank of 3.5 
to both and give the next ISD step the rank of S. 



liighty-four per cent of the total sample responded to this item. Mean 
rankings were obtained for each of the four specialties, as well as 
for the total sample and arc presented as Table 17. Table 17 indicates 
differences of opinion among thn different s»pecialties. However, there 
was universal agreement that step 5 (conduct and evaluate instruction) 
was the least important for developing an effective instructional program. 
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Step i (unalvzc svstem requirements) was seen as the most important 
step bv three of 'the four specialties. The training evaluators 
reflected their sphere of interest when they placed step 3 (develop 
objectives and tests) at the topmost position. Step 1 for the 
training evaluators was placed in the fourth position. Step 2 
(define education and training requirements) maintained a relatively 
similar position for the four specialties, that is, two groups placed 
it in second position and two placed it in third position. There 
were differences of opinion in the ranking of step 3. The training 
evaluation specialists placed step 3 in the topmost position, the 
1SD specialists ranked it in the second position, and the student 
measurement and training manager/ instructor personnel placed it 
fourth. Similarly, step 4 (plan, develop, and validate instruction) 
was regarded differently among the specialties. The training 
evaluators ranked it in the second position, to the student measure- 
ment and training managers/instructors it was in the third position, 
and it was ranked fourth by the ISD specialists. 



Table 17 

Mean Rankings of ISD Steps on Importance in Course Development 
ISD QN*33) TE (N=21) SM (N«24) TM/I (N«39) Total (N-117) 



Step 1 Step 3 Step 1 Step 1* Step 1 

Step 3 Step 4 Step 2 Step 2* Step 2 

Step 2 Step 2 Step 4 Step 4 Step 3 

Step 4 Step 1 Step 3 Step 3 Step 4 

Step 5 Step 5 Step 5 Step 5 Step 5 



tied ranks 



When the mean rankings for the total sample are considered, 
rather than the individual specialties, the order of inportance of 
each step for achieving an effective instructional program was the 
sequential order of steps 1, 2, 3, 4, 5. Since the ISD method may 
be considered as a pyramid, with each step resting on the foundation 
provided by prior steps, this rank ordering may reflect the downstream 
effects of poor performance of any individual step. From this point 
of view, poor performance of early steps would inpact on more steps 
than poor performance of later steps. 
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ii mphasi,s on ISP Steps 



The ISD steps were also evaluated with reference to the amount 
of emphasis they received during the development of the course with 
which each interviewee was most fauiliar. To this end, the paired 
comparison method was employed. The interview section titled ISD 
step comparison, presented as part of Appendix A, shows the method 
employed for collecting these data. Only 63 per cent of the entire 
sample provided usable, complete responses. The interviewee had to 
have knowledge of a course that was developed using the ISD method 
in order to respond to this item. 

The method outlined by Edwards (195 7) was used to obtain ISD 
steps emphasis scale values for each ISD step. The method requires 
calculation of the frequency with which each step was judged as 
having been given more emphasis than the other steps. The frequencies 
were then expressed as proportions (by dividing each frequency by 
the total niinber of judgments made). These proportions were corrected 
to z values (the unit normal deviate scale value which has a true 
zero point). The result of these calculations is a set of values 
along a ratio scale with a true zero point. Table 18 presents the 
results. 



Steps 1 and 2 were positioned at the bottom of the emphasis con- 
tinuum. This finding might be considered to be in line with expectation 
In many cases, step 1 (analysis of system requirements) was performed 
by a team of specialists from headquarters or an existing course was 
revised in terms of ISD, but the existing system requirements outline 
was emplo/ rt d. Similarly, step 2 was often in existence before the 
course was redeveloped. For the remaining three steps, the sequence 
was steps 3, 4, 5. These findings have implications in terms of 
development time estimates and the assignment of personnel (i.e., either 



Table 18 



Scale Values of the Emphasis the ISD Steps Received in 
Course Development 



Step Scale Value 



3 
4 
5 
1 

2 



.609 
.352 
.165 
.030 
.000 
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in terms of ability and experience or man-hours). The finding 
may also, as for the ranking data discussed above, reflect the feed 
forward characteristics of the ISD model. We note that the emphasis 
ordering, at least for the last three steps, parallels the importance 
ordering discussed above. 

In order to gain insight into the internal consistency of the 
paired comparison data, Kendall's coefficient of consistence (Edwards, 
1957) was calculated. Inconsistencies in the paired comparison 
judgments are indicated by circular triads. A circular triad would 
exist, for example, if a respondent estimated that step 3 received 
more emphasis than step 4, and step 4 received more emphasis than 
step 5, but that step 3 received less emphasis than step 5. The 
greater the number of circular triads, the more inconsistent was the 
interviewee in making these estimates. 

The coefficient of consistence can vary from 0 to 1.00; 0 repre- 
sents the maximum number of circular triads, and 1.00 represents the 
complete absence of circular triads. The coefficients of consistence 
were determined for each of the respondents. They varied from .2 to 
1.00. Of the 87 interviewees who made the paired comparisons of the 
ISD steps, 22 or 25 per cent had coefficients of consistence less than 
their judgments. The inconsistencies, as they existed in 25 per cent 
of the sample, could have been due to an indifference on the part of 
the interviewee to making careful judgments, to the difficulty in 
separating two or more very close steps (i.e., close on the continuum 
of emphasis) , or to a personality factor which would predispose one 
to being inconsistent regardless of the task on hand. 

Reliability of Interviewee Responses 

Except for the ISD specialists, two pairs of items were included 
in each interview set in order to check directly on interviewee re- 
sponse consistency. For the ISD specialists, seven pairs of items 
were so included. Re test measures, as could be obtained in a repeated 
interview, were not possible. Typically, for each reliability estima- 
tion, the two questions were asked at different points in the interview. 
Although the questions were phrased slightly differently, the expecta- 
tion was for similarity between the elicited responses. Three different 
statistical approaches were used to calculate reliability estimates. 
Where one of the involved variables was continuous in nature and the 
other variable was reduced to two categories, the biserial correlation 
coefficient was calculated. This correlation coefficient was obtained 
for those items where the ISD program was evaluated, once on a 
qualitative scale ranging from "poor" to "superior" (this scale was 
collapsed into two categories by grouping the "superior" and "very 
good" and the "good," "fair," and "poor" responses) and then on a 
quantitative scale ranging continuously from 0 to 100. 
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A coefficient of contingency was obtained where a correlation 
index was required between variables, each of which contained more 
than two classes, This was applicable where the ISD specialists 
estimated the amount of emphasis devoted to each of the ISD steps in 
the development of the course and tiicn estimated tlie relative emphasis 
given to eadi step in the paired comparison item, labeled ISD step 
comparison. 

Finally, the percentage of agreement between two items, as in the 
case where" the respondents were asked to suggest ways of improving the 
use and application of ISD and then, in another item, were asked f r 
the singlemost important improvement. Table 19 presents the item* on 
which reliability' estimates were obtained, the type of reliability 
estimate used, and the obtained reliability estimate. 



Table 10 

Reliability Estimates of Interviewee Responses 

Specialty an J Items Reliability Estimate Used Obtained Reliability 

27 and 31 Bi serial r •/* 

28 and 29 Per cent agreement 64. 
4 and Step 1* Coefficient of contingency .65 

11 and Step 2* Coefficient of contingency .61 

14 and Step 3* Coefficient of contingency .57 

17 and Step 4* Coefficient of contingency .59 
20 and Step 5* Coefficient of contingency .52 

a CO 

18 and 22 Bi serial r .59 

19 and 20 Per cent agreement 74 0 

SM , _ 

38 and 42 Hiserial r .65 

39 and 40 Per cent agreement 85* 



TM/i , c 

1 and 44 Bi serial r 

d and 42 Per cent agreement 90% 

* Tlicse data were obtained from the ISD step comparison item. 
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The hiscria! correlations obtained on the evaluation of the ISD 
program for acliieving its purposes ranged from .S5 to .78 enable 19), 
the percentage agreement between the items dealing with suggestions 
for improving tive use and application of I SI)* ranged from 64 per cent to 
90 per cent, and the coefficients of contingency** ranged from .52 to .65. 

A rule of thumb lias been applied to the minimal adequate level of 
reliability in the interview situation (Krech § Crutchfield, 194ii, pp. 260- 
262). The minimal adequate level, by this rule of thumb, for group meas- 
urement is a correlation coefficient .50. Since none of the correlation 
estimates obtained in this survey were below .50, and since the percentage 
agreement data represent correlation coefficients which are considerably 
in excess of this value, it appears that the interviewees were adequately 
consistent in their attitudes and opinions. 

ISD Evaluation as a Function of Location and Experience 

The assessment of the ISD program was examined from two additional, 
but diverse, points of view. An analysis of variance was performed to 
determine if the quantitative (on the scale of 0 to 100) evaluation of 
ISD varied as a function of: (1) Air Force base, and (2) length of time 
in the job assignment. The inclusion of the second variable was based 
on the conjecture that respondents who had been on their present job 
longest would be more resistent to change and, therefore, would. give 
lower evaluations of the ISD system. 

Four AIC technical schools were used in this analysis: Lackland, 
Sheppard, Chanute, and Lowry. Keesler Technical Training Center was 
omitted because of an insufficient sample size. Two levels of experience 
on present assignment were included. Level I consisted of respondents 
with loss than two years in their current assignment; Level II consisted 
of incumbents with more than five years in the current assignment. A 
sample of five respondents was randomly selected from each of the four 
bases, in eadi experience level. The Sheppard AFB Level II cell had 
only four ISD evaluations. The fifth evaluation was approximated by 
taking the arithmetic mean of the four evaluations. Thus, every cell 
in the analysis of variance had five entries. Tab±e 20 presents a 
siBiBitary of the results of the variance analysis. 



* The method employed for determining tiie percentage of agreement 
involved inspecting the responses to the item in which an unlimited 
number of suggestions could be made for improving ISD for a match 
with the item in which the one most i important improvement for ISD was 
identified. A match was considered to exist if the most important 
improvement suggested also appeared among the general improvements 
suggested. 

** In calculating the coefficients of contingency, the number of pluses 
that ench respondent assigned to the ISD step, as summed in the vertical 
column of the ISD step comparison matrix, represented one variable (the 
greater the numl>cr of pluses the more emphasis the step received). Hie 
scale value (0 to 100) represented the other variable. 
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Table 20 



Sunmiary of Analysis of Variance for Location and Hxperience 

Sum of Degrees of Mean 
Source of Variation Squares Freedom Square 



Location 
Experience 

Location x Experience 
Error 



269.49 
773.16 
836.99 
14553.75 



3 
1 
3 
32 



89.83 
773.16 
279.00 
451.68 



.20 
1.71 
.62 



Total 



16333.39 



39 



The variance analysis indicated that no statistically significant 
difference existed between experience levels or among locations. The 
location by experience interaction was, similarly, not statistically 
signficant. Hence, these data provide no support for a contention 
suggesting a different evaluation of the ISD system as a function of 
ATC bases or time on current job assignment. 
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DISCUSSION 



The prior chapters presented the results from the field inter- 
views conducted with the ISD, student evaluation, training manager/ 
instructor, and course evaluation specialist at five ATC centers. The 
present chapter attests to integrate some of the more salient findings 
into a logical structure and to present some of the implications of 
the findings. 

Evaluation of ISD 

liach of the four groups interviewed seemed to think that the ISD 
procedure had a different value in terms of achieving its general pur- 
pose of graduating students who meet jjb performance requirements. 
For comparative purposes, the mean nagnitude estimate, for each 
group, of the extent to which the ISD procedure achieves its goal is 
presented in Figure 5. 

Regardless of the individual mean numerical value assigned by 
the various groups, each group can be said to have regarded the ISD 
procedures favorably. This conclusion is also supported by the 
responses of the various groups when they were asked to make a 
categorical judgment oC the extent to which the ISD methods achieve 
their purpose. 

From the point of view of the general and overall effectiveness 
of the ISD procedures, it seems that they have proven their value, at 
least for the various groups involved in their inplementation. Each 
of the various groups lias had exposure to different aspects of the 
ISD procedures and, accordingly, can be anticipated to have a somewhat 
different reaction. 
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Figure 5. Mean magnitude estimates by training 
specialty (0-ICO) of ISO for achieving 
its goals. 
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Problems Within the ISP Methods 

Regardless of the general and overall positive reaction to the 
ISD methods manifested hy the groups involved, a number of problems 
within- the use and implementation of ISD became evident. 

First, there was a prevailing overcurrent of misconception about 
ISD and the need for an information dissemination program to clarify 
these misconceptions. Some of the information gaps were based on 
faulty information, e.g., "ISD will remove the instructor from the 
system and all training will be programmed." Others were less obvious 
and were related to a general lack of insight on the part of some 
supervisory personnel regarding the advantages of and the methods 
involved in the ISD procedures. While the ISD procedures have been 
presented in several formal documents, these documents have evidently 
not adiieved an indoctrinational purpose at the supervisory level. 
Accordingly, there is a need for a more general and extensive method 
for orienting supervisory personnel in the attributes and goals of 
the ISD system. 

Effective ISD implementation is also dependent on an appropri- 
ately trained corps of personnel who are thoroughly prepared in the 
performance of their individual assignments and who possess adequate 
insight into the specific aspects of other ISD requirements. It is 
believed that, in some instances, ISD aspects may have been resisted 
because the implications of the work for subsequent ISD steps were 
not fullv perceived. Accordingly, some priority seems indicated for 
attendance at the ISD course, especially for those currently concerned 
in a direct manner with ISD application. 

It is also possible that with increased training (and experience), 
the tunc required to apply the ISD procedures will decrease. Certainly, 
the interviewees bemoaned what they considered to be the excessive time 
involved in applying the various ISD procedures. This problem is 
interactive with the frequently suggested need for more personnel. It 
is also quite possible that the perceived need for an augmentation in 
the number of personnel assigned to performance of the ISD steps will 
diminish with increased proficiency in the techniques involved. 

However, wc note that t lie re is no readily accessible body of data 
regarding the amount of time required to complete the various ISD 
steps for various types of courses. Such data will be required before 
definitive statements can be made regarding the time required- to com- 
plete the ISD process for any course. 
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Tlie need for "streamlined" loll procedures was also mentioned. 
It is possible tiiat some of the time involved performing the various 
ISD steps could be shortened through tlie provision of a variety of job 
aids, work methods reorganization, organizational modification, and an 
increase in communi cat ions efficiency. Which one or ones, if any, of 
tiiese would assist most in reducing the time consumed in applying the 
various LSI) steps cannot be known as a result 0/ tlte present study. 
It is most probable that the various ISD steps would benefit differen- 
tially from the various modifications. Certainly, investigation of 
the feasibility of such solutions should take place before personnel 
allowiuiccs are au#ncntcd. 

Oi tJic other hand, if the increased training and other suggestions, 
prt rented above, for reducing the time required to apply the ISD methods 
to a course are not tenable, then the interviewee suggested increase in 
the manpower allocation seems reasonable if high priority is to be 
placed on application of the ISO method to all courses. Such universal 
application seems supportable in view of the reported degree of success 
for the ISD procedures wherever they have been applied. 

We note that the ISD model is often rejected by instructors who 
are teaching a course developed by some other model and are satisfied 
/ith tlie course and the end products it produces, it seems that some 
value may exist in a concept which advocates introducing ISD to the 
training centers first for those courses that have an immediate need 
for revision. 

There is also a need for the evaluation of how well each of the 
ISD steps is performed. There are several available analytic techniques 
that could be used for the various steps. Evaluation methods that might 
be considered for evaluating the various steps of tlie ISD model are: 
comprehensiveness and relevance diecks, statistical methods, inter- 
views and judgments, rating scales, experimental methods, checklists, 
written tests, performance tests, and task analytic matrix methods.* 
It is likely that suggestions for modifications to the application and 
utilization of ISD would result from evaluations. The introduction of 
the modifications to the model would serve to make the ISD procedures 
more useful and acceptable to application personnel. 

Finally, we note that some users have elected to implement ISD 
through performance of a selected subset of the ISD system steps. Since 
each ISD step is dovetailed with a prior step, such abbreviation does 
not seem tenable. It seems that the user should apply the total system. 
Short cutting certain required steps may serve to undermine the total 
system. 



* These tecliniques arc discussed fully in Siegel, A. I., Bergman, B.A. , 
Federman, P., f, Sell man, W.S. Handbook for the Evaluation of Tedinica l 
Training Courses and Students (AFHRL-TR-7Z-15J . ' ' 




Student Measurement Tediniques 

lTie interviewees seemed quite satisfied, on the whole, with the 
student measurement techniques employed. The criterion referenced 
checklist was especially believed to be valuable. 

However, there was little evidence that the reliability of the 
widely employed end of block written tests is determined for each test 
employed. Some of these tests seem quite short and, accordingly, 
could approach or be lower than the lowest acceptable reliability limit. 

Another end of block written test consideration is the frequency 
with which such tests are updated. It seems that updates as frequent 
as tlie end of each course may be too frequent, with the exception of 
the most obvious sorts of modification. On the other hand, updating a 
test about once a year may be too infrequent. Guidelines are required 
which will allow satisfaction of the purposes of updating a test with- 
out iu long a delay as to render the test outmoded or so short a delay 
as to require update on the basis of an insufficient data substrate. 

In some of the interviews, the item analytic data were reviewed. 
It was noted that although the required item analyses were conducted, 
the tests contained items which, at least by the usual psychometric 
standards, are not acceptable. Accordingly, it seems that item accept- 
ability psychometric guidelines are also necessary. Such guidelines 
are also contained in the above referenced Handbook (AHIRL-TR-72-15) . 

We arc aware that the time allowed for student measurement within 
a course is limited. This limitation reemphasizes the need for powerful 
test devices, where they are used. On the other hand, there seems to 
lie little gained by constricting the testing time to the point that the 
testing is not effective for achieving its purpose. Such a procedure 
represents a waste of time. Poor testing may, in fact, be worse than 
no testing at all. 

Tliere also seems to be a need for alternate or parallel forms of 
end of block wrtttcn tests for use in situations in which a student 
who fails originally is recycled and re tested. 

Additionally, there was little evidence regarding the predictive 
validity of these tests. Nor was evidence suggested regarding their 
correlation with various selection and classification tests. The 
development of such data seems mandatory. Ongoing research into new 
and improved testing methods also seems indicated. 



ERIC 





Finally, there seemed to be little reliance on the confidence 
testing and" tin? hierarchical testing approaches wiiich have come under 
some degree of acceptance in other contexts. Application of these 
concepts, at least on a trial basis, seems to be a tenable recommen- 
dation. 



Course l:\aluation Teciiniques 

Hie principal course evaluation technique employed was "critiques." 
These took the form of student critiques and critiques by graduates 
and their supervisors. These latter critiques are performed after the 
graduate has been on his technical job for a period of time. Sometimes 
the critique data are collected through questionnaire methods; less 
frequently, interviews are conducted. The questionnaire methods are 
economical and seem to produce valuable insights into areas for needed 
course revision. They neither seem to restrict unduly the range or 
freedom of response of the respondent nor do they demand an unusual 
or excessive amount of time for completion. 

There arc some comment, however, that the methods employed for 
these critiques lack objectivity. Additionally, the interpretation of 
the data yielded bv these methods was reported to be often performed by 
persons who are not completely familiar with the total subject matter 
content or with the context in which their recommendations must be 
implemented. 

Since the course evaluation constitutes the qnality control aspect 
of the ISD svstcm, a need for considerable researc 1 seems to exist in 
this sphere.' Although there is a considerable bod/ of techniques avail- 
able and although there is little, if any, doubt that some objectivity 
could be built into the already existing critiques by ad hoc methods, 
such fixes seem, at best, temporary. Accordingly, we suggest that a 
carefully designed research program be instituted in the course tech- 
niques arena. Such a program would build on the already extant state 
of the knowledge in the course evaluation sphere and develop these, for 
Air Force application, so that the course review techniques can yield 
longitudinal, objective, quantitative, continuous course review infor- 
mation. Sudi techniques, in addition to being fully synergistic with 
the ISD system, must take advantage of the most advanced knowledge in 
posttraining performance measurement and meet acceptable criterion 
requirements, e.g., reliability, objectivity, scorability, analyzability, 
practicality, predictability, etc. Such a program would also possess 
considerable implicat ions for selection and classification methodology 
and val itiation. 
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At any rate, the use of untrained course evaluation personnel 
and of untrained interviewers (as reported) for performing course eval- 
uation does not seem to constitute an acceptable practice. Interviewer 
training is relatively easy to accomplish and at least reliability, if 
not validity, can be built into the interview by such t raining . It 
also seems that such interviews should be at least seraistandardized 
and fully criterion referenced. GVe note that criterion referencing of 
the various critique forms may also be indicated.) 

Finally, there was comment that the time lag between course reviews 
and the implementation of the recommendations they elicit is often 
lengthy. This suggests a lack of direction or standardization regarding 
course update. It may also suggest a lack of coordination between 
course review personnel and those persons who are responsible for 
course update. Since course changes are implemented mostly by instruc- 
tor personnel, close coordination between these two groups seems 
manadatory if course modifications are to be implemented effectively. 

Postscript 

Anv new program, such as the ISD program, must undergo monitoring 
and a series of refinements and calibrations before it achieves its 
maximum potential. The results of the present study suggest that the 
ISD program lias achieved considerable success, and support its con- 
tinued development. The emphasis of the current set of interviews was 
on the student measurement and the course review techniques employee] 
within the ISD program. The suggestions regarding modification and/or 
refinement of certain aspects of the methods currently employed are 
aimed at improving the power of tliese methods. The performance eval- 
uation tcdinology is highly dynamic, and it is highly likely that this 
aspect of the ISD system will require ongoing modification in accordance 
with ongoing developments in the field. 
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INQUIRY INTO INSTRUCTIONAL SYSTBl DEVELOPMENT 
TEQ1NIQUES HMPLOYMLNT 

Instructional System Development 
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>ly nanv is 



f I represent 



Applied Psychological Services. Wc are under contract with the Air 
Force Human Resources Rcseardi Laboratory (AFSC) to develop a compre- 
hensive iKindlwok of training evaluation and Ftudent measurement metliods. 
The interviews we are conducting represent a part of the study involved 
in the development of this handbook. 

The purpose of tin? interview is to learn about the training eval- 
uation imd student measurement techniques being used at the various Air 
Training Command schools. In addition, we are interested in learning 
about the Instructional System Development (ISD) program and its effec- 
tiveness in the design of Air Force technical courses. The data and 
insights gained from the interviews will provide important information 
for the subsequent development of the handbook for training evaluation 
and student measurement. 

Your cooperation in the interview is greatly appreciated and will 
represent a major contribution to the study. Of course, everything you 
say will be used for the purposes of the study wily and will remain 
completely .inonymous. None of the information wc collect will be 
attributed to ;tny specific person. 
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INTRONKTION 



(.LAND INTERVIEWEE SET OF DEFINITIONS AND ISO FLOW CHART AND ALLOW A 
FEW MLMTCLS FOR HIM TO READ HEM.) 

1. QiAND I.vniRVTBvlili CARD ISP-1) Would you say that you are 
"familiar," "moderately familiar," or "unfamiliar" with the Instructional 
System Development program? 

(IF INTERVIEW!!!; IS "FAMILIAR" OR "M3DERATELY FAMILIAR," ASK QUESTIONS 
Z AND 3. IF IxrURVTEBEE IS "UNFAMILIAR" WITH ISD PROGRAM DISCONTINUE 
I NTERVI EW. ) 

1. What was the one source from which you gained most of your 
faiiiliarity witii the LSD program? 

5. (1LWD 1NTEW1EWEE CARD ISP- 3) Would you describe your depth 
of experience with ISP as "extcnrive,^ "moderate," or "slight"? 

(IF I.VriiRVllsWUli FITS ON); OF HE X-ED CELLS BELOW, CONTINUE WITH 
QUESTION 4. IF INTERVIEWliE HAS ONLY SLIGHT EXPERIENCE WITH ISD 
DISCONTINUE INTERVIEW.) 

QUESTION 1 

Fa miliar Moderately Familiar Unfamiliar 

Extensive X X 

Moderate X X 

Slight 

SPECIFIC APPLICATION 

I. ANALYZE SYSTEM REQUIREMENTS 

(SAY TO INTERVIEWEE: THE NEXT SET OF QUESTIONS WILL DEAL "SPECIFICALLY 
WITH EACH OF THE FIVE ISD STEPS. IN ANSWERING THESE QUESTIONS nilNK 
OF THE SAME COURSE YOU JUST IDENTIFIED AS * BEING MOST FAMILIAR WITH.) 

4. [HAND INTERVIEWEE CARD ISD-4) In terms of a scale which ranges 
from 0 to 100, how much emphasis would you say was placed on the ISD 
step, Analyze System Requirements, in planning the training program? 

5. Why so? 

as 
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i\v isn w.vs.i -Minus iiii.d wini a value > 20 ask questions g, 7, and 8; 

onU.RWISi; O) TO QUESTIONS 9 AND 10.) 

{>. Us in>j the same scale of values, 0 to 100, how successful was 
the analysis of system requirements step in helping to plan the course 
content? 

7. Why so? 

8. How was the 1SI) step, Analyze System Requirements, itelpful in 
planning the training program? 

9. (IF ISD WAS NOT EMPHASIZED OR EMPHASIZED VliRY LITTLE, ASK) 
Why wasn't tlie LSI) step, Analyze System Requirements, emphasized (or 
emphasized more)? 

10. Can you recall a specific situation when the ISD principle, 
Analyze System Requirements, was used and it did not really help in 
planning the training program? fell me the incident, the need, who 
fostered the use, the outcome, and why it did not help so much. 

I I. DEFINE EDUCATION OR TRAINING REQUIREMENTS 

U. (HAND INTERVIEWEE CAJ'D ISD-4) Using the scale values again, 
how -..well emphasis was placed on the ISD step, Define Education or 
Training Requirements, in planning the training program? 

12. Why so? 

13. (IF ISD WAS NOT EMPHASIZED OR EMPHASIZED VERY LITfLE, ASK) 
Why wasn't tlie ISD step, Define Education or Training Requirements, 
emphasized (or emphasized more)? 

III. DEVELOP OBJECTIVES AND TEST 

14. (HAND INTERVIEWEE CARD ISD- 4) llow much emphasis was placed 

on the ISD step, Develop Objectives and Tests, in planning the training 
program? 

15. Why so? 

16. (IF ISD WAS NOf EMPHASIZED OR EMPHASIZED VERY LITTLE, ASK) 
Why wasn't the ISD step, Develop Objectives and Tests, emphasized (or 
emphasized more)? 
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IV. PLAN, mVliUM', AM) VALIDATE INSTRUCTION 

17. (HAND lNTliRVUM;ii CARD ISD- 4) How much enphasis was placed 
oil the I Si) principal, Plan, Develop, and Validate Instruction, in 
developing and implementing the course? 

18. Why so? 

19. (IF ISD WAS NOT EMPHASIZED OR EMPHASIZED VERY LITrLE, ASK) 
Why wasn't the ISD step, Plan, Develop, and Validate Instruction, 
emphasized (or emphasized more)? 



v. qwHuer and evaluate instruction 

20. (HAND INTERVIEWEE CARD ISD- 4) Using the scale values, 0 to 
100, what was the extent to which the step, Conduct and Evaluate 
Instruction, was emphasized in inplementing and controlling the 
effectiveness of the course? 

21. Why so? 

(IF ISD WAS EMPHASIZED WITH A VALUE > 20, ASK QUESTIONS 22, 23, and 
24; MllERKISE GO TO QUESTIONS 25 AND 26.) 

22. Using the scale again, how successful is the ISD step, Conduct 
and Evaluate instruction, for implementing and controlling the effec- 
tiveness of the course? 

23. Why so? 

24. ilow was this ISD step of conducting and evaluating instruction 
helpful in implementing and controlling the effectiveness of the course? 

25. (IF ISD WAS NOT EMPHASIZED OR EMPHASIZED VERY UTILE, ASK) 
Why wasn't this ISD step emphasized (or emphasized more)? 

26. Can you think of a sDccific situation when the ISD step, 
Conduct and Evaluate Instruction, was applied and it did not prove io 
be -beneficial in implementing and controlling the course? - Tell m the 
incident, the need, who fostered the use, the outcome, and why it did 
not help so much. 
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GENERAL OPINION 



27- in general, would vou rate or describe the ISD program as 
"superior," "very good," "good," "fair," or "poor" for achieving the 
stated objectives of turning out graduates who meet job performance 
requi renients efficient y and effectively? 

.'8. What specific or general suggestions c J o you have for improving 
the use and application of ISD? 

2i>. What one improvemciil would help ISD utilization most? 

30. In your opinion, why do some people avoid ISD utilization? 

31. (11AXD INTERVIEWEE CARD ISD-31) I tow would you evaluate the ISD 
program, on the basis of values 0 to 100, for achieving its purposes? 

32. (HAND INTERVIEWEE ANSWER SHEET) Please assign rank orders, 
from 1 to 5, in this column (POINT J so bat we can determine which ISD 
steps you think help most to achieve an. effective instructional program. 
The number one is the best, number 5 represents the worst. If you want 
to tie two steps, then assign both the average value between those two 
positions and skip two nunfoers. For example, if you wanted to rank ISD 
steps 1 and 4 as tied for third position, you would assign the rank of 
3.S to both and give the next ISD step tJie rank of 5. 

33. What do you think are the three greatest problems in using the 
ISD model? 

54. is there anything else you would like to tell me? 
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Familiar ; Know and understand all of the fSD steps 

Moderately familiar : Know and understand most, but 
not all, oi tlic ISD steps 

Unfamiliar : Do not know any of the ISD steps 



Handout Card for Item 1 



Kx tensive: Used and applied all the ISD steps 

Moderate : Used and applied most, but not all, of 
the ISD steps 

Slight: Mover used or applied any of the ISD steps 



Handout Card for Item 3 



Scale of lii«phasis 



0 10 20 30 40 50 60 70 80 90 100 
Very Little Moderate lixtensive 



Ilan.'out Card for Item 4 



Scale of Achievement 



0 10 20 30 40 50 5«1 70 80 90 100 
Poor Fair Good Very Good Superior 



Handout Card for Item 31 
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INQUIRY INTO INSTRUCTIONAL SYSTEM DEVELQPMjNT 
TEONIQUES EMPLOYMENT 

Training Evaluation 
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My iuu.k* is _ „„_ t 1 represent 

Applied Psychological Services. We are under contract with the Air Force 
Human Resources Research Lilxmitory (AFSC) to develop a comprehensive 
handbook of training evaluation and student measurement methods. The 
interviews we are conducting represent a part o r the study involved in 
tlu* development of this handbook. 

The purpose of the interview is to learn about the training eval- 
uation and student measurement techniques being used at the various Air 
Training itamuuid schools. In addition, wc arc interested in learning 
about the Instructional System I level jpment (ISD) program and its effec- 
tiveness in the design of Air Force technical courses. The data and 
insights gained from the interviews will provide inqrartant information 
for the subsequent development of the handbook for training evaluation 
and student measurement. 

Your cooperation in tiie interview is great iy appreciated and will 
represent a major contribution to the study. Of course, everything 
you say will be used for the purposes of the study only and will remain 
completely anonymous. None of the inton ation wc collect will be 
attributed to any specific person. 
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{SAY IX) INTLRVHMj.: Wli IliiFIM: TRAINING EVALUATION AS illi- wrriiRMi NA- 
TION of- now wi;ll u;akxi\g oiuhcitvls kkri; inotifim), m- i-fficiilncy of 
nu ; isu froclss, and nil: profichlncy of graduates in performing \m jobs 
HiR wiiiai -nuiY ki;ri; trai^li*. such evaluation often leads to course 

REVISION. ANSWER THESE QUESTIONS IN 'lliRMS OF IlIE COURSE WITH WIIQI YOU 
SAID YOU ARE MOST FAMILIAR.) 

1. Was tlie course periodically updated to meet dumping requirements? 

2. (IF YES) How often? 

?>. (IF YES) Where did tlie information indicating a needed course 
revision coiiie from? 

4. (IF Yi~S) How did the information get fed back to you? 

5. What sorts of course review techniques were used, as separate 
from field evaluation? 

u. Would vou rate the course review tediniques (as separate from 
field evaluation) used for obtaining information that can be used for 
modifying the course as "superior," "very good," "good ," "fair," or 
"poor ?" 

7. What was tlie principal problem with the course review tech- 
niques employed? 

8. Which course review techniques best identified areas of over 
and under training? 

9. Tell me. about a specific incident in which course review results 
were actually used? Tell me the incident, the need,, who fostered the 
use, the outcome, and why it was helpful. 

10. Were field evaluations conducted for tlie purpose of updating 
the course contents? 

11. (IF YiiS) How often? 

12. (I< ; YES) Mint types of field evaluation tediniques arc (were) 
u-ed? 

13. (IF YES) How did this need get fed back to you? 

14. Would you rate tlie field evaluation techniques used for obtain- 
ing information that can be used for modifying the course as "superior," 
"very good," "good," "fair," or "poor?" 
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15. What was the principal problem with the field evaluation tech- 
niques employed? 

16. Which field evaluation technique best identified areas of over 
and under training? 

17. Tell me about a specific incident in which the field evaluation 
results were actually used by the course developers. Tell me the incident, 
the need, who fostered the use, the outcome, and why it was helpful. 

18. In general, would you rate or describe the ISD program as 
?T?°i> "very good," "good," "fair," or "poor" for adiieving the 

stated objectives of turning out graduates who meet job performance 
requirements efficiently and effectively? 

19. What one improvement would help ISD utilization most? 

20. What specific or general suggestions do you have for improvinc 
the use and application of ISD? . 

21. In your opinion, why do some people avoid ISD utilization? 

11 ■ (iiAMD I.VTLRVIBVLi: CARD Tli-22) ifow would you evaluate the ISD 
program, on tlxc basis oi the values 0 to 100, for achieving its purposes? 

23. QLVVD INTLRVIBV^ ANaviiR aOlhT) Please assign rank orders, from 
I to 5, in this column (iviM) so that we can determine which ISD steps 
you think help most to achieve an effective instructional program. The 
number one is the jest, number 5 represents the worst. If you want to 
tie two steps, then assign both the average value between those two 
positions and skip two numbers. For example, if you wanted to rank ISD 
steps 1 and 4 as tied fur third position, you would assign the rank of 
j. 5 to botn and give the next ISD step the rank of 5. 

ISD model? 1131 d ° ***** *** three ^ rcatest P rob lems in using the 
25. Is there anything else you would like to tell me? 
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Scale of Achievement 



0 10 20 30 40 50 60 70 80 90 100 
Poor Fnir Good Very Good Superior 



Handout Card for Item 22 
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INQUIRY INTO INSTRUCTIONAL SYSTLM DEVULOMiNT 
TiiONIQUCS OTLOYM-NT 

Student Measurement ', 
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tfy turn.- is . 1 represent 

0 i i i « i ■ — ^ — ■ " ' 

Applied Psychological Services. i\ic are under contract with the Air 
Force Human Resources Research Laboratory (AFSC) to develop a compre- 
hensive handl>ook of training evaluation and student measurement methods. 
The interviews we are conducting represent a part of the study involved 
in the development of this handlwoR. 

The purpose of the interview is to learn about the training eval- 
uation and student measurement tcduiiques being used at the various Air 
Training Command schools. In addition, we are interested in learning 
about the lust met ional System Development (ISD) program and its effec- 
tiveness in the design of Air Force technical courses. The data and 
insights gabled from the interview, -'ill provide important information 
for the subsequent development of the handbook for training evaluation 
and student measurement. 

Your cooperation in the interview is greatly appreciated and will 
represent a major contribution to the study. Of course, everything 
you say will be used for the purposes of the study only and will 
remain completely anonymous, .'tone of the information we collect will 
bo attributed to any specific ptirtM. 
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(SAY IX) INTliRVIiaViili: ML HAVli SLVLRAL QUESTIONS ON STUDHNT MP.ASURIM1NT 
TLQINIQULS. m HiFIMi SltfllLVT MLASURDfiiNT AS MIASURII3 FOR Tllli PURPOSli 
OF EVALUATING INDIVIDUAL SJUDLNT AGilUVlMiNT AND PROGRESS, ANSWER 'YlUISli 
QUESTIONS IN TliRMS OF Tltti QKIRSii WITH WUIQ1 YOU SAID YOU ARC MOST 
FAMILIAR.) 

1. (SiOV CARD SM-1) IVIiat are (were) tlie different types of student 
measurement (student achievement) techniques employed? 

2. On the basis of end of block written test results, how did you 
know if a student passed or failed or how well he did? 

(SAY 1X3 LViiiRVIliWlili: THINK ABOUT HO- MuIHODS EMPLuYLD FOR CONSTRUCTING 
LN'D OF BLOCK WRITON TUSTS) . 

3. Was a formal item analysis conducted? 

4. (IF YLS) iiow so? 

5. Was any attempt made to correlate the end of block test scores 
with other measures of performance? 

6. (IF YES) How so? 

7. Why do you tliink that these end of block written tests were 
effective for evaluating students? 

8. How so? 

9. What was the principal problem, in your opinion, in using these 
written tests? 

10. How so? 

11. Were the students informed of their end of block written test 
results? 

12. (IF YHS) What recommendations were made to those students who 
do not achieve satisfactory results? 

13. Were work sample tests employed? 

(IF YtS, ASK QUESTION'S 14, IS, 16, 17, 18, 19, and 20; OTriliRWISli GO TO 
QUESTION 21) 

14. ilow did you know, on the basis of the work sample test scores, 
whether or not a student passed or failed or how well he did? 



ERIC 



92 

97 



BEST COPY AVAILABLE 



15. Why do you think work sample tests were effective for measuring 
student achievement? 

16. How so? 

17. Wliat was the principal problem, in your opinion, with the work 
sample tests? 

18. How so? 

10. Were students informed of their work sample test results? 

20. (IF YliS) What recommendations were made to those students who 
did not aciiieve satisfactory results? 

21. Were criterion checklists employed? 

(IF YliS, ASK QUESTIONS 22, 23, 24, 25, 26, 27, and 28; OliOWISii CO TO 
QUiibTION 29) 

22. On the basis of the criterion cliecklist, how did you know if a 
student passed or failed or how well he did? 

23. Why do you think criterion checklists were effective for measuring 
student achievment? 

24. iiow so? 

25. tVhot was the principal problem, in your opinion, with the 
criterion l. ..cklists? 

26. How so? 

27. Were students informed of their criterion checklist results? 

28 (IF YliS) What recommendations were made to those students who 
did not aciiieve satisfactory results? 

29. Were there any student measurement techniques not used in the 
course that you would have like to see used? If so, what are they? 

30. How were the results of student evaluations fed back to the 
course developers? 

31. iiow were the student evaluations used by the course developers? 
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32. Can you tell me of a specific incident where the course developers 
took some correction action on the basis of student measurement results? 
Tell me the incident, the need, who fostered the use, the outcome, and 
why it was helpful. 

33. How frequently were student measurement techniques reviewed? 

34. How frequently were student measurement techniques modified as 
a result of these reviews? 

35. i/id the students fill out critique forms on the course? 

36. (IF YES) Who reviewed these critiques? 

37. Can you tell me a specific incident where the course developers 
took some corrective action on the basis of the critique forms filled 
out by the students? Tell me the incident, need, who fostered the use, 
the outcome, and why it was helpful. 

38. In general, would you rate or describe the ISD program as "super io: 
"very good," "good," "fair." or "poor" for achieving the stated objectives 
of turning out graduates who meet job performance requirements efficiently 
and effectively? 

39. What specific or general suggestions do you have for improving 
the use and application of ISD? 

40. What one improvement would help ISD utilization most? 

41. In your opinion, why do some people avoid ISD utilization? 

42. (HAND INTERVIEWEE CARD SM-42) How would you evaluate the ISD 
program, on the basis of the values 0 to 100, for achieving its purposes? 

43. (HAND INTERVIEWEE ANSWER SHEET) Please assign rank orders, from 
1 to 5, in this column (POINT) so that we can determine which ISD steps 
you think help most to achieve an effective instructional program. The 
number one is the best, number S represents the worst. If you want to 
tie two steps, then assign both the average value between those two 
positions ard skip two numbers. For example, if you wanted to rank ISD 
steps 1 and 4 as tied for third position, you would assign the rank of 
3.5 to both and give the next ISD step the rank of 5. 

44. What do you think are the three greatest problems in using the 
ISD model? 

45. Is there anything else you would like to tell me? 
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End of block written test 
Criterion Checklist 
Programed instruction test 
Instructor's rating 
Oral test 

Work sample performance test 



llandout Card for Item 1 



Scale of Achievement 

0 10 2 0 30 40 50 6 0 70 80 90 100 
Poor Fair Good Very Good Superior 



llandout Card for Item 42 
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INQUIRY INTO INSTRUCTIONAL SYSTEM DEVHLOPMiNT 
TECHNIQUES EMPLOYMENT 

Training Managers/Instructors 
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My name is • I represent 

Applied Psychological Services. Wc are under contract with the Air Force 
Human Resources Research Laboratory (AFSC) to develop a comprehensive 
handbook of training evaluation and student measurement methods. The 
interviews, wc arc conducting represent a part of the study involved in 
the development of this handbook. 

The purpose of the interview is to learn about the training eval- 
uation and student measurement techniques being used at the various Air 
Training Command schools. In addition, we are interested in learning 
about the Instructional System Development (ISD) program and its effec- 
tiveness in the design of Air Force technical courses. The data and 
insights gained from the interviews will provide important information 
for the subsequent development of the handbook for training evaluation 
and student measurement. 

Your cooperation in the interview is greatly appreciated and will 
represent a major contribution to the study. Of course, everything you 
say will be used for the purposes of the study only and will remain 
completely anonymous. None of the information we collect will be 
attributed to any specific person. 
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1. In general, would you rate or describe the ISD program as 
M supcrior t " 'Very good." "good." "fair," or "poor" for achieving the 
stated objectives of turning out graduates who meet job perfonn.*jice 
requirements efficiently and effectively? 

2. (HAND INTERVIEWEE CARD TM/I-2) I would like you to select from 
the adjectives that appear on this card the three that you tliink best 
describe the LSD model. 

5. (11A\D INTERVIEWER C/\RD TM/I-5) Now I would like you to do the 
same with this list of adjectives. 

4. What would you say were the three best aspects of the TSD model? 
Rank order these, e.g., 1 « best, 3 * least best. 

5. What would you say were the three worst aspects of the ISD 
model? Rank order these, e.g., 1 - worst, 3 = least worst. 

b. What changes would you like to see introduced that would improve 
the ISD program? 

7. Why do you support the employment of ISD? 

8. What kinds of problems do course developers typically find 
when developing new ISD courses? 

9. What sorts of criteria do you have for judging instructional 
effectiveness? 

10. Do these criteria differ for different courses? 

11. (IF YES) How so? 

(SAY TO INTERVIEWEE: NOV WE HAVE QUESTIONS ON TRAINING EVALUATION. 
WE DEFINE TRAINING EVALUATION AS THE DETERMINATION OF IKT.V WELL LEARNING 
OBJECTIVES WERE IDENTIFIED, THE EFFICIENCY OF THE ISD PROCESS, AND THE 
PROFICIENCY OF GRADUATES IN PERFORMING THE JOBS FOR WHICH THEY WERE 
TRAINED. SUai EVALUATION OFTEN LEADS TO COURSE REVISION. ANSWER THESE 
QUESTIONS IN TERMS OF THE COURSE WITH MHO! YOU SAID YOU ARE MOST 
FAMILIAR). 

12. Was the course periodically updated to meet clianging requirements 

13. (IF YES) (low often? 

14. (TF YES) Where did the information indicating a needed course 
revision crane from? 

15. (IF YES) How did the information get fed back to you? 
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10. What sorts of course review techniques were used, as separate 
from ficlU evaluation? 

17. Would vou rate the course review techniques (as separate from 
field evaluation! used for obtaining information that can be used for 
modifying the course as "superior," 'Very good," "good," " fair t or 

"voor?" 

18. What was the principal problem k a . the course review 
tcdiniques employed? 

19. Which course review technique best identified areas of over 
and under training? 

20. Tell me about a specific incident in which the course review 
results were actually used. Tell me tht incident, the need, who 
fostered the use, the outcome, and why it helped so much. 

21. Were field evaluations conducted for the purpose of updating 
the course contents? 

22. (IF YES) How often? 

23. (IF YES) What types of field evaluation techniques are (were) 
used? 

24. (IF YES) 1 low did this need get fed back to you? 

7 5. (IF YES) Would you rate the field evaluation techniques used 
for obtaining information that can be used for modifying the course as 
"superior," "very good," "good," "fair," or ^orT 

26. (IF YliS) What was the principal problem with the field eval- 
uation techniques employed? 

27. (IF YES) Which field evaluation technique best identified 
areas of over and under training? 

28. (IF YES) Tell me about a specific incident in which the field 
evaluation results were actually used by the course developers. Tell me 
the incident, the need, who fostered the use, the outcome, and why it 
helped so much. 

(SAY TO INTERVIEWEE: WE HAVE SEVERAL QUESTIONS ON STUDENT MEASUREMENT 
TECINIQUES. WE DEFINE STUDENT MEASUREMENT AS MEASURES FOR THE PURPOSE 
OF EVALUATING INDIVIDUAL STUDENT ACHIEVEMENT AMD PROGRESS. ANSWER THESE 
QUESTIONS IN TERMS OF THE SAME COURSE WE WERE DISCUSSING.) 
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29. What are (were) the different types of student measurement 
(student acnievemcnt; techniques employed? 

30. i>n the b:isis of end of block written test results, how did 
you know if a student passed or failed or how well he did? 

31. Why do you think that these end of block written tests were 
effective for evaluating students? 

32. How so? 

53. Are the students informed of their end of block written test 
results? 

54. (IF YLS) What recommendations are made to those students who 
do not get satisfactory evaluations? 

35. How were the results of student measurements fed back to the 
course developers? 

36. liow were the student measurements used by the course developers 

37. Can you tell me of a specific incident where the course devel- 
opers took some corrective action on the basis of student measurement 
results? Tell me the incident, the need, who fostered the use, the 
outcome, and why it was so helpful. 

38. Did the students fill ouc critique forms on the course? 

39. (IF YliS) Who reviewed these critiques? 

40. Can you tell me a specific incident where the course developers 
took some corrective action on the basis of the critique forms fillea 
out by the students? Tell me tlie incident, the need, who fostered the 
use, the outcome, and why it was so helpful. 

41. What specific or general suggestions do you have for improving 
the use and application of ISD? 

42. What one improvement would help ISD utilization most? 

43. In your opinion, why do some people avoid ISD utilization? 

44. (HAND LNTHRVIEWH- CARD TM/I-44) How would you evaluate the 
ISD program on the basis of the values of 0 to 100 for achieving its 
purposes? 
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45. (HAND LV rLRVIUWIU: ANSWER S11HLT) Please assign rank orders, 
from i to 5 in this columi (MINT) so that we can determine which ISD 
steps help most to achieve an effective instructional program. The 
number one is the best, number 5 represents the worst. If you want to 
tic two steps, then assign both the average value between those two 
positions and skip two numbers. For example, if you wanted to rank 
ISD steps 1 and 4 as tied for third position, you would assign the rani 
of 3.5 to both and give the next ISD step the rank of 5. 

46. Wiiat procedures have been instituted to evaluate the worth of 

ISD? 

47. Which aspects of ISD do you like best? Why? 

48. Which aspects o*" ISD do you like least? Why? 

49. Is there anything else you would like to tell me? 



helpful 
appropriate 
conpletc 
good 

profitable 

organized 

efficient 

excellent 

effective 

dynamic 

tlandout Card for Item 2 



mundane 
unworkable 
poor 
static 

unsuccessful 

weak 

limited 

unsuitable 

tedious 

costly 

Handout Card for Item 3 



Scale of Achievement 



0 10 20 30 40 50 60 70 80 90 100 



iiandout Card for Item 44 



Poor 



Fair 



Good 



Very Good 



Superior 
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1. In general, would you rate or describe the ISD program as 
"superior," "very good," "good," "fair," or "poor" for achieving the 
stated objectives of turning out graduates who meet job performance 
requirements efficiently and effectively? 

2. What specific suggestions do you have for improving the use 
and application of ISD? 

3. What one improvement would help ISD utilization most? 

4. In your opinion, why do some people avoid ISD utilization? 

5. (HAND INTERVIEWEE CARD HQ- 5) I would like you to select from 
the adjectives that appear on this card the three that you think best 
describes the ISD model? 

0. (HAND INTERVIEWEE CARD H3-6) Now I would like you to do the 
same with this list of adjectives. 

7. What would you say were the three best aspects of the ISD 
model? Rank order these, i.e., 1 « best, 3 * least best. 

8. What would you say were the three worst aspects of the ISD 
model? Rank order these, i.e., 1 « worst, 3 = least worst. 

9. What changes would you like to see introduced that would 
improve the ISD program? 

10. Why do you support the employment of ISD? 

11. What kinds of problems do course developers typically find 
when developing new ISD courses? 

12. (tiAND INTERVIEWEE CARD HQ- 12) How would you evaluate the ISD 
program on the basis of the values of 0 to 100, for achieiving its 
purposes? 

13. What procedures have been instituted to evaluate the worth 
of ISD? 

14. What sorts of criteria does headquarters have for judging 
instructional effectiveness? 

15. Do these criteria differ for different courses? 

16. (IF YES) How so? 

17. Which aspects of TSD do you like best? Why? 



18. Which aspects of ISD do you like least? Why? 

19. Do you Ivave any plans or desires to terminate the usage of ISD? 

20. (IF YiiS) How so? 

21. What arc tlie three most significant problems headquarters staff 
srsonnel have in using ISD? 

Is there anything you would like to tell me? 
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helpful 
appropriate 
conplete 
good 

profitable 

organized 

efficient 

excellent 

effective 

dynamic 



Handout Card for Item S 



mundane 
unworkable 
poor 
static 

unsuccessful 

weak 

limited 

unsuitable 

tedious 

costly 



Handout Card for Item 6 



Scale of Achievement 



0 10 20 30 40 50 60 70 80 90 100 
Poor Fair Good Very Good Superior 



Handout Card for Item 12 
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isn srur cwarisons 

TIkj purpose of this rating form is to determine how much each ISD 
step was emphasized (relative to every other step) in achieving the 
ultimate purpose of graduating students who meet job performance require- 
ments. The five ISD steps to be rated for emphasis in achieving the 
purposes of the course are arranged around the chart below. Thinking 
only about the same course that was identified in the interview as the 
one with which you were most familiar, start by comparing ISD step 1 
(across) with ISD step 2 (down). If you think that step 1 (across) was 
emphasized more than step 2 (down), place a plus sign (♦) in step l's 
column (alongside step 2). If you think that step 1 (across) was 
emphasized less than step 2 (down), place a minus sign (-) in step l's 
coltmn (alongside step 2). 

In the example below, three different instructional methods were 
rated for meeting the objectives of the course. The objectives were 
designated by the letters A, B, and C. The judge, starting by comparing 
A with B, decided that A was more effective than B. Thus, he placed a 
plus sign in A's column (alongside B). In the next judgment, he compared 
A with C and decided that A was more effective than C. A plus sign was 
thus placed in A's column (alongside C). In comparing B with C, the 
judge decided that B was loss effective than C and therefore a minus 
sign in B's column (alongside C). 
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In completing the chart below, work down each column by comparing 
step 1 with steps 2, then step 3, through step 5. Next, work down 
column 2 by comparing step 2 with step 3 through 5. Continue the 
process of comparing pairs of ISD steps until the chart is completed 
(a total of 10 comparisons). 
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RESPONSE SHEETS 
Training Evaluation 

Interviewee's Name Location 

Current Assignment I low Lcng pate 

ISD Course and Number Most Familiar With 

(CROSS OUr "ISD" IF COURSE DID NOT HAVE IsD MflbEL AFPUH) TO I T) 

1. yes no 

2. at the end of each course q uarterly semi-annually 

yearly every other year other (specify) 

3. 

4. written report m eeting word of mouth other (specify) 

5. student interviews i nstructor interviews instructor suggestions 

review of field requirements Other (specify) 

6. superior very good g ood fair p oor 

7. 



8. 
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******* mitABu: 

9. Incident ______ 

Need 

Press 

Outcome 

Why helpful 

10. yes no 

11. at the end of each course q uarterly semi-annually yearly 

once every other year other (specify) 

12. graduate evaluations supervisor's evaluations of graduates 

graduate's critique of training received other (specify) 



13. written report m eeting word of mouth Other (specify) 



14. superior very good good fair poor 

15. 



16. 



17. Incident^ 
Need 
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Press 



Outcome 



Why helpful 
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Bis* 



18. 
19. 



superior 



vor>' good 



good 



fair 



jioor 



20. (R-19) 



22. (R-18) 

23. (R) ISP Steps- Rank Order 

Plan, Develop, and Validate Instruction 

Define Education or Training Requirements 

Conduct and Evaluate Instruction 

Analyze S/stem Requirements 

Develop Objectives and Tests 

24. (1) 

(2) 

(3) 

25. 
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